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BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 


This project has been funded, either wholly or in part, with federal funds from the U.S. 
Department of Labor, Employment and Training Administration under contract number 
DOLJ101A21513. The contents of this publication do not necessarily reflect the views or 
policies of the U.S. Department of Labor, and mention of trade names, commercial projects, 
or organizations does not imply endorsement of them by the U.S. government. 


Overview 


Making the successful transition to adulthood has become increasingly difficult for many young peo- 
ple in the United States, particularly for those without a college education. Those without a high school 
degree face even tougher prospects, with especially high unemployment rates and falling wages. A 
typical worker without a high school diploma earns less today than the same worker did in the 1970s. 
YouthBuild is a program that attempts to improve prospects for less-educated young people, serving 
over 10,000 individuals each year at over 250 organizations nationwide. Each organization provides 
hands-on, construction-related or other vocational training, educational services, case management, 
counseling, service to the community, and leadership-development opportunities, to low-income 
young people ages 16 to 24 who did not complete high school. 


YouthBuild was evaluated using a randomized controlled trial, in which eligible young people at par- 
ticipating programs were assigned to either a program group, invited to enroll in YouthBuild, or a 
control group, referred to other services in the community. The evaluation included 75 programs 
across the country and nearly 4,000 young people who enrolled in the study between 2011 and 2013. 
This report, the final in the evaluation, presents the program’s effects on young people after four years. 


Main Findings 


The effects observed through four years indicate that the program provides a starting point for redi- 
recting otherwise disconnected young people, but one that could be improved upon. 


e YouthBuild increased the receipt of high school equivalency credentials. 


e YouthBuild increased enrollment in college, largely during the first two years. Very few young 
people had earned a degree after four years, and the program had a very small effect on degree 
receipt. 


e YouthBuild increased survey-reported employment rates, wages and earnings, but did not in- 
crease employment as measured with employer-provided administrative records, which might not 
include certain kinds of employment, such as jobs in the gig economy and other types of informal 
work. 


e YouthBuild increased civic engagement, largely via participation in YouthBuild services. It had 
no effects on other measures of positive youth development. 


e YouthBuild had few effects on involvement with the criminal justice system. 


e As with many youth programs, YouthBuild’s benefits through four years do not outweigh its 
costs. But it is too early to draw firm conclusions about YouthBuild as an investment, since the 
benefits accrue over participants’ lifetimes. 


YouthBuild has continued to evolve since it started in the 1970s and even since the study began. The 
findings from the evaluation can inform its future direction and help it have greater impact. 
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Executive Summary 


Finding a good job has become increasingly difficult for young people without a college educa- 
tion. Young adults were the hardest hit by the Great Recession, and even today their unemploy- 
ment rates remain high.' Those without a high school degree face even tougher prospects, with 
especially high unemployment rates and falling wages. A typical worker without a high school 
diploma earns less today than the same worker did in the 1970s, highlighting the importance of 
postsecondary education or training in today’s economy.’ 


YouthBuild is a program that tries to improve these prospects for less-educated young 
people. It started in the 1970s with one program in New York City, which set out to engage oth- 
erwise disconnected young people in the improvement of their community through the renovating 
and building of housing for low-income residents. At the same time, the program provided par- 
ticipants with the leadership opportunities, education, and job training skills they needed to suc- 
cessfully navigate the transition from adolescence to adulthood. Over the next several decades, 
YouthBuild expanded dramatically. Today, operating with both federal and private funding, there 
are over 250 YouthBuild programs nationwide, serving over 10,000 young people each year. 
Many of the programs are part of the YouthBuild Affiliated Network, under the umbrella of 
YouthBuild USA, which provides technical assistance, funding, and program design standards to 
its members. 


Each YouthBuild program provides hands-on, construction-related or other vocational 
training, educational services, case management, counseling, service to the community, and lead- 
ership-development opportunities, to low-income, out-of-school young people ages 16 to 24. As 
the program has grown, it also changed somewhat to reflect the changing economy. The voca- 
tional training component, for example, while still focused primarily on construction, has ex- 
panded to other areas. And the educational services, still largely high school equivalency prepa- 
ration, have expanded to include charter schools offering high school diplomas. In addition, 
programs are increasingly focused beyond high school credentials and on postsecondary education. 


In accordance with federal legislation, the U.S. Department of Labor (DOL) is a major 
funder of YouthBuild, providing grants to about 70 programs each year through a competitive 
review process. In 2010, DOL, with initial support from the Corporation for National and Com- 
munity Service (CNCS), awarded a contract to MDRC and its partners Social Policy Research 
Associates and Mathematica Policy Research to conduct an impact evaluation of YouthBuild. 


U.S. Department of Labor, Bureau of Labor Statistics, “Labor Force Statistics from the Current Population 
Survey” (website: data.bls.gov/timeseries/Ins14024887, 2017a). 

?Economic Policy Institute, “State of Working America Data Library, Wages by Education” (website: 
www.epi.org/data/#?subject=wage-education, 2017). 
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The evaluation includes 75 programs across the country, receiving funding from either DOL 
or CNCS, and nearly 4,000 young people who enrolled in the study between 2011 and 2013. 
The evaluation examines YouthBuild’s effects on the young people it serves, assessing effects on 
a wide range of outcomes, including education and training, work and earnings, youth develop- 
ment, and involvement in the criminal justice system. 


This report, the final in the evaluation, presents effects of the program after four years 
and shows that YouthBuild had positive effects on some important outcomes. The program led 
to a sizable increase in high school equivalency credential receipt. The program also increased 
college enrollment, although it had only a very small effect on degree receipt. YouthBuild also 
led to an increase in employment and earnings at the four-year point, as reported on the survey, 
but there were no significant effects on work reported to the unemployment insurance system. 
The program increased civic engagement but did not have effects on other measures of youth 
development or on rates of involvement with the criminal justice system. 


A cost analysis shows that YouthBuild involves a substantial upfront investment per par- 
ticipant, owing to its relatively small size, its location in large urban areas, and the educational 
and training services it provides. A limited benefit-cost assessment shows that YouthBuild is val- 
uable to participants, but its net value to taxpayers and society will depend on the size of earnings 
impacts beyond the four years covered by the evaluation. 


It is hard to say whether the effects on work and earnings will grow over time. The effects 
observed so far suggest that the model provides a good starting point, but one that will need to be 
improved upon if it is to make large and sustained impacts on the young people it serves. The 
increased emphasis on postsecondary education, for example, has been important, and Youth- 
Build successfully increased access and attendance. However, similar to many low-income stu- 
dents around the nation, YouthBuild participants need additional support to stay in school and 
complete a degree. Providing those services may be outside of YouthBuild’s scope, but the pro- 
gram could do more to partner with supportive postsecondary institutions. Similarly, more work 
needs to be done to increase employment and earnings, although finding what works on the em- 
ployment side for this population has been a challenge for many programs. The program’s recent 
focus on apprenticeships and skills training programs may be one strategy. 


3Jennifer Ma and Sandy Baum, Trends in Community Colleges: Enrollment, Prices, Student Debt, and Com- 
pletion (New York: The College Board, 2016); Susan Scrivener, Michael J. Weiss, Alyssa Ratledge, Timothy 
Rudd, Colleen Sommo, and Hannah Fresques, Doubling Graduation Rates: Three-Year Effects of CUNY’s Ac- 
celerated Study in Associate Programs (ASAP) for Developmental Education Students (New York: MDRC, 
2015). 
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The YouthBuild Model 


The YouthBuild model includes a mix of education, vocational training, counseling, leadership 
development, and service to community, all provided within a culture that emphasizes respect for 
young people and positive youth development. Eligibility is typically limited to out-of-school 
young people ages 16 to 24 who have dropped out before completing high school and who meet 
one of the following criteria: they are from low-income or migrant families, are current or former 
foster youth, are involved with the criminal justice system, are disabled, or are children of incar- 
cerated parents. 


Programs recruit or rely on word of mouth to identify interested applicants, who then go 
through assessments before enrolling, such as tests of basic skills and one-on-one interviews. 
Most frequently, programs then implement a rigorous Mental Toughness Orientation, which can 
last from a single day to several weeks. Mental Toughness Orientation is designed to facilitate 
group bonding and ready recruits for the program’s activities. It also serves as a period when 
many young people are screened out because they stop attending or otherwise fail to follow es- 
tablished rules. 


Most young people who make it through Mental Toughness Orientation enroll in Youth- 
Build, are offered the program’s services, and participate for 6 to 12 months. New participants 
typically begin the program with a group of other enrollees, and that group alternates between 
educational and vocational training. The components of the model are intended to be integrated 
and are designed to be offered together. 


YouthBuild’s services consist of the following: 


e Educational services such as instruction in basic skills, remedial education, 
and alternative education leading to a high school diploma or high school 
equivalency credential. A growing number of programs also offer services to 
prepare young people for enrollment in postsecondary education, such as col- 
lege tours, assistance with financial aid applications, and, in some cases, dual 
enrollment. 


e Vocational training, typically direct hands-on training in construction, in 
which participants rehabilitate or build housing for low-income people. In 
2012, certain DOL-funded programs were authorized to provide a “construc- 
tion plus” model, in which training is offered for in-demand occupations other 
than construction. Before that date, some programs were already providing 
training for other vocations such as certified nursing assistant, commercial 
driver, or information technology professional through non-DOL funding. 
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e Youth development services, including leadership training and service to 
community. Defining features of YouthBuild, these services are addressed in 
multiple ways and serve multiple purposes. Leadership training is approached 
through structured curricula or formal and informal roles within the Youth- 
Build program that participants may play, such as on committees, in the class- 
room, on work sites, or in community activities and meetings. Young people 
serve their community by constructing affordable housing and through other 
activities; this service both addresses the community’s needs and provides op- 
portunities for young people to practice leadership and other skills. 


e Supportive and transitional services include counseling, case management, 
life skills training, workforce preparation, follow-up services for up to one 
year, stipends for participation, and other forms of support, such as help with 
transportation, child care, or housing. All of these services are designed to help 
young people address challenges that may prevent them from achieving suc- 
cess in the program or beyond. 


The Evaluation 


The YouthBuild evaluation uses a random assignment research design, in which eligible young 
people at participating programs around the country were assigned to either a program group, 
which was invited to enroll in YouthBuild, or to a control group, which was not able to enroll in 
the program but was provided with information on other services in the community. The research 
team selected a mix of programs receiving funding from DOL and from CNCS in 2011 for the 
evaluation. From the 74 programs that received grants from DOL in 2011, 60 were randomly 
selected to participate in the study, and 58 were ultimately able to participate. From the 24 pro- 
grams that received CNCS grants but not DOL grants in 2011, 17 were selected as suitable to 
participate in the study. 


These 75 programs (58 funded by DOL and 17 not funded by DOL but receiving funding 
from CNCS) enrolled a total of 3,929 young people into the study between August 2011 and 
January 2013, a number that exceeded available slots. For each enrollment cycle, each program 
used its typical selection process to create a pool of applicants deemed eligible and appropriate 
for YouthBuild. These applicants were then assigned at random to fill the available program slots 
or to a control group. In most programs, random assignment took place before Mental Toughness 
Orientation. 


The evaluation consisted of three components. First, a process study consisted of in-per- 
son visits by members of the research team to nearly all participating programs, to examine im- 
plementation and operations on the ground and to hear firsthand the perceptions and experiences 


ES-4 


of the participating young people and staff members. Second, an impact study tracked the pro- 
gram and the control groups for four years using survey data and administrative records to exam- 
ine the program’s effects on a wide range of outcomes, including educational enrollment and 
attainment, work and earnings, involvement in the criminal justice system, family structure, and 
social and emotional development. Finally, a cost study estimated the costs of operating and run- 
ning YouthBuild and compared these costs with the potential benefits achieved. 


The analysis presented in this report is based on several data sources. First, the research 
team administered surveys to a random subset of study participants 12, 30, and 48 months after 
they entered the study. These surveys collected information on education and training, work, fam- 
ily formation, youth development, involvement in the criminal justice system, and child support. 
Second, the team obtained administrative records for the full study sample on employment and 
earnings (from the National Directory of New Hires) and postsecondary enrollment (from the 
National Student Clearinghouse). Program staff members provided the team with data for the cost 
study during the process study visits. 


The young people enrolled in the study generally fit the profile of typical YouthBuild 
participants. The majority of study participants were male (64 percent) and most were either black 
(63 percent) or Latino (15 percent). On average, study participants were nearly 20 years old when 
they entered the study. Over 90 percent did not have a high school diploma or equivalency cre- 
dential when they entered the study, and about 60 percent of them had left school after completing 
the tenth or eleventh grade. 


Recap of Earlier Findings 


The findings from the process study, presented in an earlier report, indicated that there was vari- 
ation in how programs implemented the components of the model, in response to their local con- 
texts. The program model was not highly prescriptive and was designed to allow variation based 
on local circumstances. Overall, however, the participating programs implemented the Youth- 
Build model well and faithfully. Fidelity to the YouthBuild model among programs was most 
consistent and highest in vocational services and varied more in leadership development and post- 
secondary educational preparation. 


Findings from the interim report presented effects through 30 months. That report indi- 
cated that about 75 percent of young people assigned to the program group went on to participate 
in YouthBuild during the first year after study entry. About half of these young people reported 
that they graduated from YouthBuild. Those assigned to enroll in the program had higher rates of 
participation in educational, training, and personal development activities than those who were 
assigned to the control group. This was in spite of the fact that control group members also had 
relatively high rates of participation in other educational and training activities located in the com- 
munity, indicating that those who seek to enroll in YouthBuild are a relatively motivated group. 
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Even with this motivated sample, YouthBuild increased high school equivalency credential re- 
ceipt, college enrollment, and participation in vocational training through Month 30. YouthBuild 
also led to a small increase in employment rates during the second year after participants entered 
the study, and a small increase in self-reported wages and earnings 30 months after study entry. 
Finally, the program increased the rate at which young people engaged in volunteer work, but 
had few effects on other measures of youth development or involvement in the criminal justice 
system. 


Findings Through Four Years 


e YouthBuild increased the receipt of high school equivalency credentials. 


Most young people entered the program without having completed high school, and 
YouthBuild led to a sizable increase in high school equivalency credential receipt. By the 30- 
month point, about 18 percent of young people in the control group reported having earned this 
credential, compared with 31 percent of young people in the program group, for an increase of 
14 percentage points. By the 48-month point, about 24 percent of the control group had earned 
the credential and the program impact was 11 percentage points. (See Table ES.1.) This effect 
was estimated for all young people in the program group and did not account for the fact that 25 
percent of program group members did not participate in YouthBuild after study enrollment. The 
effect on young people who did participate was about 15 percentage points. 


e YouthBuild increased enrollment in college, largely during the first two 
years. It had a very small effect on degree receipt. 


While many YouthBuild programs focus their educational services on attainment of high 
school equivalency credentials, a growing number of programs have an explicit focus on helping 
young people transition to postsecondary education. About 13 percent of young people in the 
control group reported having enrolled in college since study entry, and YouthBuild enrollment 
led to an increase in that rate of 8.6 percentage points. The effect for young people who actually 
participated in YouthBuild was higher, at about 12 percentage points. Most young people who 
attended college did so at two-year colleges and most of the program’s impact was on attendance 
at two-year institutions. Effects on enrollment were larger at YouthBuild programs with strong 
postsecondary educational services. 


However, less than 2 percent of the study sample reported earning a degree of any type 
by 48 months, and the program had a statistically significant, but very small effect on that out- 
come. A look at enrollment over time shows that the program’s impacts on college attendance 
occurred largely during the first two years. After that point, enrollment rates for young people in 
the program group fell, and the program impact had faded to zero. 
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Table ES.1 


Impacts on Key Outcomes at 48 Months 


YouthBuild Control Difference 
Outcome Group Group (Impact) 
Education and training (%) 
Earned high school equivalency credential 34.5 23.5 11.0*** 
Ever enrolled in vocational school 32.9 21.7 11.2*** 
Received trade license/training certificate 5.3 3.4 1.9% 
Ever enrolled in postsecondary courses? 21.3 12.7 8.6*** 
Ever received a postsecondary degree* 1.7 0.9 0.8** 
Work and earnings 
Currently employed (%) 50.9 46.4 4.5** 
Average weekly earnings ($) 206.7 174.1 32.6*** 
Average earnings in Year 4° ($) 6,980 6,729 251 
Youth development 
Civic engagement’ (%) 94.3 90.6 3.7*** 
Self-esteem score 3.3 3.3 0.0 
Criminal justice involvement (%) 
Arrested since random assignment 32.0 31.3 0.7 
Convicted since random assignment 19.8 17.4 2.5 
Sample size (total = 2,721) 1,784 937 


SOURCES: MDRC calculations using data from the National Directory of New Hires (NDNH), the National 
Student Clearinghouse, and responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

@Postsecondary education outcomes are based on data from the National Student Clearinghouse. The 
sample size for these outcomes is 3,929 (2,700 program group members and 1,229 control group mem- 
bers). 

>This earnings outcome is based on data from the NDNH. Social Security numbers were unavailable for 
some sample members, who therefore could not be matched to the database. The sample size for these 
outcomes is 3,878 (2,662 program group members and 1,216 control group members). 

“Civic engagement” is defined as at least one of the following: volunteering, being registered to vote at 
the time of the survey, having voted, or having been involved in politics or local community activities. 

‘Self-esteem is measured using the 10-item Rosenberg Self-Esteem scale. Response categories range 
from 1 = “strongly disagree” to 4 = “strongly agree,” where higher scores indicate higher levels of self-es- 
teem. Responses to the 10 items are averaged. 


ES-7 


e YouthBuild increased survey-reported employment rates and wages and 
earnings, but did not increase employment as measured with administra- 
tive records. 


The opportunities for education and training in YouthBuild should help participating 
young people find jobs after completing the program, and the early impacts the program had on 
high school equivalency credential receipt and college enrollment support this claim. Although 
the findings span four years of follow-up, they still offer an early glimpse at the potential careers 
of the participating young adults, who were on average 24 years old at the four-year point. The 
interim report documented that the program led to an increase in unemployment insurance sys- 
tem-reported employment rates in Year 2 of about 3 percentage points, and an increase in survey- 
reported weekly earnings of about 12 percent. At the 48-month survey, 46.4 percent of the control 
group reported working, compared with 50.9 percent of young people in the program group, for 
an impact of 4.5 percentage points. In addition, young people in the program group earned higher 
wages (more of them eamed at least $10 per hour), with the result that they had 19 percent higher 
weekly earnings. 


Similar increases in work were not found using the National Directory of New Hires 
records data, which included employer-reported quarterly earnings covered by the unemployment 
insurance system, although there was a trend over the four-year follow-up period toward positive 
earnings impacts. Surveys and records data often show different results, particularly for low-in- 
come groups who are more likely to have informal jobs, be self-employed, participating in the 
gig economy or work for employers who may not report their wages. Separate analyses suggest 
that part of the reason for the lack of effects in the records data was that these data did not capture 
some self-employment and informal or intermittent work among the study sample. Another rea- 
son for the lack of effects using the records data was that they were estimated for the full study 
sample, and YouthBuild’s effects on work were somewhat more positive for survey respondents 
than for the full sample. 


Finally, effects on work were larger for less academically prepared young people. First, 
effects were larger for young people who had left high school in the earlier grades. Similarly, 
impacts on work were larger among those programs that opted not to screen out applicants based 
on low basic education scores. 


e YouthBuild increased civic engagement, largely via participation in 
YouthBuild services. It had few effects on other measures of positive 
youth development. 


Participation in YouthBuild may lead to a number of positive changes for participants 
through effects on education, work, and leadership opportunities. By the 48-month point, the pro- 
gram had led to a large increase in reported rates of volunteering, of about 21 percentage points, 
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much of which occurred while young people were participating in YouthBuild given its strong 
emphasis on service to community. In contrast, YouthBuild had no effects on other measures of 
youth development, such as self-esteem, self-efficacy, future orientation, and feelings of social 
support. Other research on youth programs suggests that it is difficult to create lasting changes in 
many of these attitudinal measures.* 


e YouthBuild had few effects on involvement with the criminal justice sys- 
tem. 


Finally, by the 48-month point, just over 30 percent of the young people in the study 
reported that they had been arrested since they entered the study, which was not much higher than 
the rate reported at the 30-month point. Just under 20 percent had been convicted, most commonly 
for either drug or property offenses. The program had no effect on arrest or conviction rates. 


e As with many youth programs, YouthBuild’s benefits through four years 
did not outweigh its costs. But it is too early to draw firm conclusions 
about YouthBuild as an investment, since the benefits accrue over partic- 
ipants’ lifetimes. 


YouthBuild involves a significant upfront investment in the young people it serves. A big 
part of that investment comes from DOL, but the programs themselves must match 25 percent of 
their DOL grants with non-federal funds. The total cost per YouthBuild participant, from all fund- 
ing sources, is above average relative to other youth programs. One reason for the higher cost is 
the fact that YouthBuild programs, which typically serve 30 to 40 young people per cohort, are 
smaller than other youth programs, which limits the scale economies than can be achieved. Other 
reasons include the cost of providing services to obtain a high school credential, the cost of provid- 
ing construction training, and the prevalence of programs in large urban areas, where services 
tend to be more expensive. Despite its high relative cost, a partial benefit-cost analysis shows that 
YouthBuild comes closer to breaking even after four years than the two other youth programs 
used for comparison. After four years, however, only a fraction of the potential return has been 
measured. More time is needed to see if the effects observed to date translate into lasting eco- 
nomic improvement. 


Conclusion 


The effects observed so far indicate that the program provides a starting point for redirecting 
otherwise disconnected young people, albeit one that could be improved upon. The findings 


4Jacquelymne Eccles and Jennifer Appleton Gootman, “Features of Positive Developmental Settings,” pages 
86-118 in Jacquelynne Eccles and Jennifer Appleton Gootman (eds.), Community Programs to Promote Youth 
Development (Washington, DC: National Academy Press, 2002). 


show, for example, that YouthBuild successfully served as an access point to college for disad- 
vantaged young people, but that the next steps are to increase this impact and to increase persis- 
tence and degree receipt. While it is arguably beyond YouthBuild’s service reach to directly affect 
college persistence, there may be more it can do to provide post-program services or to partner 
with supportive postsecondary educational institutions. The study findings also suggest that pro- 
grams may want to conduct less screening based on academic preparation, since program effects 
on work were larger for young people with lower educational levels. However, the DOL perfor- 
mance standards, outcomes used to rate all programs receiving DOL funding, may discourage 
programs from doing so. Programs might avoid serving young people with the lowest expected 
outcomes, even though they may be the group for whom the program makes the most difference. 


In terms of effects on employment, as the labor market has continued to change, many 
programs have expanded their vocational training to areas beyond construction. DOL has also 
recently emphasized the placement of graduates into Apprenticeship programs. It is difficult to 
assess how much the Great Recession affected the observed effects on work. The follow-up pe- 
riod for the report was during a time when many sectors were still recovering, and youth unem- 
ployment rates remained high, particularly for black and Latino young men.° The collapse of the 
housing market also affected not only employment opportunities in construction, but the quality 
of the training available to young people while they were in the program. 


Finally, although not measurable with the impact data, the implementation report docu- 
mented that funding stability was an important issue that affected program quality. Most programs 
were heavily reliant on the DOL funding cycle, in which they competed for and hopefully re- 
ceived a new grant every two years. Not winning a new grant often led to staff and program 
instability. One potential area for program improvement may be to introduce a longer funding 
cycle, in order to provide more stability to existing programs. 


YouthBuild is a very different program than it was when it began in the 1970s, and it has 
even evolved in the seven years since the evaluation started. The findings from the evaluation 
show that YouthBuild led to a number of positive effects on the young people it served, and they 
can inform its direction in the future and help it have greater impact. 


°U.S. Department of Labor, Bureau of Labor Statistics, “Employment Status of the Civilian Noninstitutional 
Population 16 to 24 Years of Age by School Enrollment, Age, Sex, Race, Hispanic or Latino Ethnicity, and 
Educational Attainment” (website: www.bls.gov/web/empsit/cpseea16.htm, 2017b). 
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Chapter 1 


Introduction 


Changes in the labor market over the past several decades have made it tougher for many Amer- 
ican workers to find and maintain well-paying jobs. Young people have been hit especially hard 
by these changes. Unemployment for individuals ages 16 to 24 increased the most during the 
Great Recession, peaking at just over 19 percent in late 2009.! In July 2017, the share of all 16- 
to 24-year-olds with employment during the summer months was 55 percent. While this percent- 
age has come up since falling to a low point in 2010, it is still much lower than the rate of 65 
percent that prevailed in 2000.2 The unemployment rate for young people without high school 
diplomas is especially high, at just over 15 percent in late 2017.? Rates of unemployment are 
higher for Latino and African-American young men.‘ High rates of youth unemployment are a 
concern given that early problems in the labor market can have lasting effects,° and given that 
unemployment rates have dropped substantially for other segments of the U.S. population. 


As a result, it has become more and more difficult for many young people to make the 
transition to adulthood successfully. Those without high school diplomas — and there are ap- 
proximately three million of them today — face particular challenges. One study documented 
that among young people without a diploma and from low-income families, under a third went 
on to earn either a high school diploma (11 percent) or a GED certificate (19 percent) within eight 
years after they were originally scheduled to graduate.* A GED credential by itself has generally 
not been found to have much worth in the labor market, although there is evidence that it may 
lead to earnings increases in the longer term for some groups.” A GED certificate can also be a 
route to postsecondary education, although a minority of GED credential recipients go on to enroll 
in college and even fewer complete degrees, *® even though postsecondary training is increasingly 
viewed as a necessary step on the path to a good job.° Finding ways to reengage these young 
people in education and work and help them become thriving adults is one of our nation’s central 
social policy challenges. 


U.S. Department of Labor, Bureau of Labor Statistics (2016a). 
U.S. Department of Labor, Bureau of Labor Statistics (2016b). 
3U.S. Department of Labor, Bureau of Labor Statistics (2017b). 
4U.S. Department of Labor, Bureau of Labor Statistics (2017b). 
°Neumark (2002). 

Hurst, Kelly, and Princiotta (2004). 

7Heckman, Humphries, and Mader (2010). 

®Heckman, Humphries, and Mader (2010). 

°Hurst, Kelly, and Princiotta (2004). 


YouthBuild is one program that attempts to help this group. YouthBuild is a federally 
and privately funded program operated by over 250 organizations nationwide, serving over 
10,000 young people each year. Each organization provides construction-related training or other 
vocational training, educational services, counseling, and leadership development opportunities 
to low-income, out-of-school young people ages 16 to 24. The vast majority of programs provide 
construction training as their vocational instruction; in these programs participants work on ren- 
ovating or constructing housing for low-income or homeless people. YouthBuild distinguishes 
itself from other programs serving young people without high school diplomas through a program 
environment that emphasizes youth development and leadership, capitalizing on participants’ 
strengths, and empowering participants to take responsibility for their lives. 


In 2010, the U.S. Department of Labor (DOL), with additional support from the Corpo- 
ration for National and Community Service (CNCS), awarded a contract to MDRC and its part- 
ners, Social Policy Research Associates and Mathematica Policy Research, to conduct a random 
assignment impact evaluation of YouthBuild. The evaluation includes 75 programs across the 
country and nearly 4,000 young people who enrolled in the study between 2011 and 2013. 


This report is the final one for the evaluation. '° It presents the program’s effects on young 
people four years after they entered the study. The report examines effects on a range of outcomes, 
including participation in education and training, educational attainment, youth development, 
civic engagement, work and earnings, and involvement in the criminal justice system. 


In sum, YouthBuild led to a number of promising initial effects. The program led to a 
sizable increase in high school equivalency credential receipt and college enrollment, although it 
had only a very small effect on degree receipt. YouthBuild also led to an increase in employment 
and earnings at the four-year point, although only in work reported on survey measures. The pro- 
gram increased civic engagement, particularly volunteering, but had no effects on other measures 
of youth development or on involvement in the criminal justice system. A limited benefit-cost 
assessment shows that YouthBuild is valuable to participants, but its net value to taxpayers and 
society depends on the size of earnings impacts beyond the four years covered by the evaluation. 


YouthBuild 


YouthBuild started in the late 1970s with one program in East Harlem, New York City, called 
the Youth Action Program (YAP). YAP allowed young people to improve their community by 
renovating and building housing, while at the same time giving them the education and job train- 


10Rarlier reports documented implementation at the 75 programs in the study (Wiegand et al., 2015), and 
provided impact estimates 30 months after participants entered the study (Miller et al., 2016). 


ing they needed. In the late 1980s and early 1990s, other programs modeled on YAP were devel- 
oped under the name “YouthBuild.” To support these replication efforts, staff members from 
YAP founded YouthBuild USA in 1990 to provide technical assistance and training to new 
YouthBuild programs. In 1992, under the umbrella of YouthBuild USA, a number of local Youth- 
Build programs came together to form the YouthBuild Affiliated Network, made up of programs 
that agreed to uphold certain standards for performance and program design and to support advo- 
cacy efforts on behalf of the program and low-income young people. 


The expansion of the program was initially supported by private grants and then later by 
the federal government. In the early 1990s, federal legislation allocated funds to be granted to 
YouthBuild programs through an annual, competitive process under the auspices of the U.S. De- 
partment of Housing and Urban Development. In 2006, responsibility for YouthBuild was trans- 
ferred to DOL’s Employment and Training Administration. Each year, DOL awards grants to 
about 70 programs, based on a competitive review process that assesses past performance and 
community needs. The grants typically range in size from $700,000 to $1,100,000 and are in- 
tended to cover two years of program services for one or more cohorts of young people and 12 
months of follow-up services.'! Grantees are also required to raise nonfederal funds to match 25 
percent of the DOL grants they receive. 


The YouthBuild network also receives funding from a variety of other public and private 
sources through grants to YouthBuild USA. For example, about 70 YouthBuild programs nation- 
wide receive annual funding from CNCS through its grant to YouthBuild USA. These programs, 
called YouthBuild AmeriCorps programs, strongly emphasize service to the community and post- 
secondary enrollment.!* Other sources of funding include state appropriations, educational fund- 
ing (based on average daily attendance), and foundation grants, among others. 


The programs across the country are quite diverse in structure and size. Some programs 
are community-based organizations or faith-based organizations, while others are run by local 
government agencies or educational institutions. In addition, at least 40 YouthBuild programs are 
now able to grant high school diplomas within their respective states. While the average program 
serves between 30 and 40 young people per year, some are quite small, serving 8 to 10 young 
people, while others serve 75 or more per year. !° 


‘A “cohort” is a group of participants who join a program at the same time and move through it together. 
At the time of the evaluation, the grants covered two years of program services plus 9 to 12 months of follow-up 
services. 

*A distinguishing feature of YouthBuild AmeriCorps programs is that participants are eligible for education 
awards when they complete YouthBuild. These awards range from about $1,175 to $5,500 depending on partic- 
ipants’ hours of service and other activities. 


3One program in the evaluation served more than 200 young people each year. 


The Model 


The YouthBuild model includes a mix of education, vocational training (typically in construc- 
tion), counseling, leadership development, and service to the community. Eligibility is usually 
limited to out-of-school young people ages 16 to 24 who have dropped out before completing 
high school and who meet one of the following criteria: they are from low-income or migrant 
families, are in or have aged out of foster care, are involved with the criminal justice system, are 
disabled, or are children of incarcerated parents. 


Programs recruit or rely on word of mouth to identify interested applicants, who then go 
through assessments before enrollment such as tests of basic skills and one-on-one interviews. 
Most programs then implement a rigorous Mental Toughness Orientation (MTO), which can last 
from a single day to several weeks. MTO is designed to facilitate group bonding and ready re- 
cruits for the program’s activities. It also serves as a period when young people are screened out, 
sometimes as an explicit goal of the program, because they stop attending or otherwise fail to 
follow established rules. 


Most young people who make it through MTO enroll in YouthBuild, are offered the pro- 
gram’s services, and participate for 6 to 12 months. New participants typically begin the program 
with a cohort of other new young people, and the cohort alternates between education and voca- 
tional training. For example, many programs use a weekly rotation in which young people par- 
ticipate in education one week and vocational training the next. The components of the model are 
intended to be integrated and are designed to be offered together, which distinguishes YouthBuild 
from other youth programs that may offer some of the same services. 


YouthBuild’s services consist of: 


e Educational services such as instruction in basic skills, remedial education, 
and alternative education leading to a high school diploma or equivalency cre- 
dential. A growing number of programs also offer services to prepare young 
people for postsecondary education. 4 


e Vocational training, typically direct hands-on training in construction in 
which participants rehabilitate or build housing for low-income people. Begin- 
ning in 2012, certain DOL-funded programs were authorized to provide a 
“construction plus” model, in which training is offered for in-demand occupa- 
tions outside construction. Before that date, some programs were already 


14 fforts to promote college enrollment have been supported by several grants to YouthBuild USA and are 
also authorized and encouraged by DOL. 


providing training for other vocations such as certified nursing assistant, com- 
mercial driver, or information technology professional using non-DOL funds. 


e Youth development services, including leadership training and service to the 
community. These services are defining features of YouthBuild that are ad- 
dressed in multiple ways and serve multiple purposes. Leadership training is 
approached through structured curricula or formal and informal roles for par- 
ticipants within the YouthBuild program on committees, in the classroom, on 
work sites, or in community activities and meetings. Young people provide 
service to the community by constructing affordable housing and through 
other activities; this service to the community attends to the community’s 
needs, teaches the value of helping others, and provides opportunities for 
young people to practice leadership and other skills. 


e Supportive services and transition services include counseling, case man- 
agement, life skills training, workforce preparation, follow-up services for up 
to one year, stipends for participation, and other forms of support, such as help 
with transportation, child care, or housing. All of these services are designed 
to help young people address challenges that may prevent them from achiev- 
ing success in the program or beyond. 


Its focus on youth development distinguishes YouthBuild from more traditional employ- 
ment programs for young people. YouthBuild reflects a movement to empower young people, 
advocate for them, foster their civic engagement and activism, and encourage them to take on 
roles of responsibility and leadership in their personal lives and broader communities. 


Together, the combination of services is hypothesized to create a number of positive 
changes for participants, which are shown in the rightmost boxes in Figure 1.1. In the short term, 
YouthBuild aims to increase participants’ basic skills and help them earn a high school equiva- 
lency credential or high school diploma. Young people in the program can also accumulate work 
experience at the work sites, earn training credentials, and find jobs. Less tangibly, YouthBuild 
aims to stimulate lasting changes in attitudes that will keep young people on a positive trajectory 
and increase their civic engagement. In the longer term, YouthBuild aims to see its participants 
enroll in and complete college, maintain stable employment, earn more money, and have less 
involvement than their peers in the criminal justice system. 


The bottom of Figure 1.1 shows that a variety of contextual factors can influence a par- 
ticipant’s experience in YouthBuild and subsequent outcomes. A number of program features 
could also affect outcomes and impacts — chiefly the program’s fidelity to the YouthBuild 
model. This report examines whether programs of varying fidelity have different effects, and 
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whether YouthBuild has different effects on participants with different characteristics. Given 
YouthBuild’s focus on educational attainment, for example, it may be that participants with 
higher educational levels at entry are better able to take advantage of the services offered by the 
program. This report therefore examines YouthBuild’s effects based on the educational level of 
participants. 


Apart from fidelity to the YouthBuild model, a range of other factors might also be asso- 
ciated with outcomes, such as the intensity of MTO, the extent to which programs focus on prep- 
aration for and connections to postsecondary enrollment, or the length of time the program has 
been operating YouthBuild. A formal analysis of how such program features are associated with 
programs’ impacts is presented in Chapter 4 of this report. 


There have been a number of studies of YouthBuild over the past 20 years, although none 
were conducted as randomized controlled trials.'° Most studies have either documented program 
implementation or tracked YouthBuild graduates to assess how they fare after leaving the pro- 
gram. Ferguson, Clay, Snipes, and Roaf, for example, conducted a formative evaluation from 1991 
to 1994 of the first five YouthBuild replication programs, documenting the challenges programs 
faced in achieving high fidelity to the model, and the essential features of the model that help 
young people succeed. Another study of YouthBuild graduates, selected primarily from a subset 
of established YouthBuild programs, found that YouthBuild graduates fared relatively well after 
leaving the program, with a majority either enrolled in school or training, or working and earning 
above a certain wage per hour.’ 


The Evaluation 


The present evaluation of YouthBuild uses a random assignment design, in which eligible young 
people at participating programs around the country were assigned either to a program group, 
invited to enroll in YouthBuild, or to a control group, who were not allowed to enroll in Youth- 
Build (for a two-year period) but instead were provided information on other services in the com- 
munity. 


The evaluation consists of three components: 


e Process study. This study examines the operations of the YouthBuild pro- 
grams in the evaluation and the perceptions and experiences of the participat- 


'SSee, for example, Ferguson, Clay, Snipes, and Roaf (1996); Hahn, Leavitt, Horvat, and Davis (2004); 
Mitchell et al. (2003); Hahn and Leavitt (2007); and Cohen and Piquero (2009). 


'6Hahn, Leavitt, Horvat, and Davis (2004). 


ing young people. The study assesses each program’s fidelity to the Youth- 
Build model. Findings from the process study were presented in an earlier re- 
port.?” 


e Impact study. This study tracks both the program and the control groups for 
four years using survey data and administrative records. '* The impact analysis 
examines the program’s effects on a wide range of outcomes, including enroll- 
ment in education and educational attainment, work and earnings, involvement 
in the criminal justice system, family structure, and social and emotional de- 
velopment. This report presents the final findings from the impact study, with 
effects being measured a full four years after participants enrolled in the study. 


e Cost study. This analysis estimates the costs of operating and running Youth- 
Build and compares these costs with any positive benefits that were achieved. 
The findings from this study are reported in Chapter 5 of this report. 


Program Selection 


The programs selected for the evaluation include a mix of those receiving funding from DOL and 
from CNCS in 2011. The first selection pool included the group of 74 programs that received 
grants from DOL. Three of these programs were deemed unsuitable for the study and were 
dropped from the pool.'? From the remaining 71 programs, 60 programs were randomly selected 
to participate in the study.*° After discussions with program staff members and DOL, 2 of the 60 
selected programs were subsequently dropped from the evaluation.*! 


The remaining programs were selected from a group of programs that did not receive 
DOL funding in 2011 but received relatively large grants from CNCS, through its National Direct 


'”Wiegand et al. (2015). 

18 A dministrative records are data collected primarily for the management of programs and public services. 

19mterviews with staff members at these three programs indicated that young people assigned to a control 
group would be likely to receive services that were very similar to YouthBuild services, which would provide a 
poor test of YouthBuild’s effects. These programs accounted for only 4 percent of the expected YouthBuild 
enrollment among all grantees. 

?0Budget considerations prohibited selecting all 71 programs for the study. Programs were selected using 
probability-proportional-to-size sampling, in which larger programs representing more young people had a 
greater probability of selection. Selecting programs in this way meant that each program slot, or young person, 
had an equal chance of selection. 

*1Several discussions with program and DOL staff members revealed that random assignment was not fea- 
sible at these two programs because they would not be able to enroll study participants during the evaluation’s 
intake period. 


Grant to YouthBuild USA.*? DOL and CNCS chose to include the CNCS-funded programs in 
the evaluation in order to examine whether DOL-funded programs have different impacts than 
CNCS-funded programs.” Of the 24 programs that received relatively large CNCS grants, 7 were 
deemed not suitable for the evaluation, leaving 17 CNCS-funded programs in the study.** 


In total, 75 programs were selected for the study: 58 DOL-funded programs and 17 
CNCS-funded programs. Although effects are examined across all programs combined, it is im- 
portant to remember that the group of CNCS-funded programs is not a random sample of such 
programs. However, as mentioned later, the programs in the evaluation look very similar to the 
broader population from which they were selected. 


Program and Study Intake 


The participating programs enrolled 3,929 young people into the study between August 2011 and 
January 2013.*° The research team worked with each program to implement random assignment 
during one or more of its enrollment cycles during this period. The general procedure was to 
determine the number of suitable applicants for the program and offer available program slots to 
these young people through a lottery-like process. The programs used their normal selection pro- 
cesses as much as possible to create the eligible pool of applicants. Figure 1.2 illustrates a typical 
YouthBuild selection process, though the details varied slightly from program to program. Ran- 
dom assignment would then occur among that smaller, eligible pool of applicants, somewhere 
between eligibility determination and enrollment. 


As shown in the figure, the first step in intake was recruitment, which is typically a major 
activity for most programs. It involved considerable staff effort, in part because YouthBuild pro- 
grams usually recruited many more applicants than they needed to fill their open slots. Excess 


*2Discussions with DOL and YouthBuild USA staff members suggested that the study should draw from 
those programs that received grants of $95,000 or more from CNCS. Of the 40 programs that received CNCS 
funding but not DOL funding in 2011, 24 programs received grants of $95,000 or more. 

*3This analysis was ultimately not conducted, given that the funding source distinction is not very clear cut. 
Many of the DOL-funded programs also received CNCS grants, for example, and several of the 17 CNCS- 
funded programs in the study received DOL grants in the subsequent year. 

*4Four of the programs planned to shut down in the coming year, and three indicated that young people in 
the control group would be likely to receive services similar to YouthBuild services. 

*SThree of the 75 programs were unable to conduct random assignment during the intake period because 
their enrollment numbers were too low. These programs were included in the process study but are not included 
in the impact study. 
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NOTES: CNCS = Corporation for National and Community Service 

“These are the eligibility criteria required by the funders. Programs may use additional criteria. 

>Although YouthBuild is a program aimed at high school dropouts, 25 percent of participants in programs funded by the U.S. Department of 
Labor can have a high school diploma or not be in one of the target populations, as long as they are deficient in basic skills or have been referred 
to a high school diploma-granting YouthBuild program by another high school. 


applicants were needed because some applicants were determined to be ineligible or unsuitable 
for the program during screening, and others dropped out during screening. See Chapter 4 of the 
earlier implementation report for more details on recruitment and screening. *° 


After recruitment, the next step was to determine whether a young person met additional 
eligibility requirements beyond those listed above. Staff members reviewed young people’s ap- 
plication forms and conducted various assessments and interviews to determine whether appli- 
cants met these additional requirements. Most commonly, applicants had to live within certain 
geographical boundaries and have minimum math and reading scores on assessments of basic 
skills. 


At some point during the screening process, the majority of study programs administered 
an academic skills test, usually the Test of Adult Basic Education (TABE) or the Comprehensive 
Adult Student Assessment Systems (CASAS). Programs were divided in how they used these 
tests: Roughly half used them to screen out applicants who did not meet minimum test score 
requirements, and the rest used them for diagnostic purposes, to establish a baseline for each 
applicant’s skills and to determine what academic services that applicant needed.?”? Requirements 
for minimum test scores reflected a tension within the program between trying to reach and enroll 
more “at-risk” young people and needing to meet specific performance targets for educational 
and employment outcomes established by DOL for its youth programs. Put most simply, the 
lower a young person’s test score, the more they might need the services YouthBuild provides, 
but the less likely they might be to obtain targeted outcomes. How grantees balanced these com- 
peting priorities determined how they used the results of these skills tests. Some programs used 
additional criteria, such as not having a substance abuse problem, to determine whether a young 
person was appropriate for the program. Applicants who met these criteria were often described 
as having demonstrated “readiness,” or a motivation and capability to make positive changes in 
their lives. 


After the initial screening process, young people were invited to MTO. As a result of the 
intensive application and screening process, nearly half of the young people who turned in appli- 
cations to the study programs did not receive invitations to MTO, either because they dropped 
out during the screening process or because the program decided they were not suitable or ready 
for YouthBuild. 


MTO is designed to determine young people’s willingness to change, to gauge their in- 
terest and motivation, to build teamwork while they get to know one another, and to introduce 
them to the specifics of the YouthBuild program. The duration and intensity of MTO varied quite 


*6Wiegand et al. (2015). 


?7Programs that required a minimum score set that minimum at a sixth-grade reading level, on average. See 
Wiegand et al. (2015). 


11 


a bit; on average, it lasted for 10 days and for seven hours per day. According to staff members 
at the study programs, the top four activities conducted during MTO were team-building exer- 
cises, life skills training, leadership development and community service, and academic work, all 
of which are also activities in which young people participated after officially enrolling in Youth- 
Build.’° Although it is an important step in the process, DOL does not count young people who 
do not complete MTO, and thus do not go on to receive core services, as program participants. 


MTO also served as an additional form of screening. During MTO, staff members might 
determine that a young person was not ready for YouthBuild or not a good fit for the program 
and ask that person to stop attending, or a young person might stop attending and therefore self- 
select out of participation. An average of one in four recruits who were invited to MTO did not 
complete it.?° 


An important issue for the research team was the timing of random assignment at each 
program. Would young people deemed eligible and appropriate for YouthBuild be randomly as- 
signed to the program or the control group before MTO, sometime during MTO, or after they had 
completed MTO? One argument for conducting random assignment before MTO was that many 
staff members considered this orientation to be an important part of the YouthBuild program. 
Conducting random assignment before the orientation would ensure that young people assigned 
to the control group did not receive a part of YouthBuild. However, an argument for conducting 
random assignment after MTO was that many young people drop out of this orientation and do 
not go on to receive YouthBuild’s core services. Conducting random assignment before the ori- 
entation would therefore also mean that many young people assigned to the program group would 
never receive core YouthBuild services, which would hinder the study’s ability to detect program 
impacts. Ultimately, the decision about when to conduct random assignment was made by each 
program, with input from the research team. Most programs (81 percent of programs, representing 
75 percent of study participants) opted to conduct random assignment before MTO or during its 
first few days.°° 


In order to ensure an adequate number of young people for available slots, the research 
team aimed for a random assignment ratio in which 60 percent of eligible young people would 
be assigned to the program group and 40 percent to the control group. In practice, most programs 
had difficulty securing enough excess applicants to meet this 60:40 ratio and were allowed to 


*8Wiegand et al. (2015). 

*°wiegand et al. (2015). 

°The timing of random assignment was found to be unrelated to the percentage of young people in the 
program group who ultimately went on to receive the core YouthBuild services. Similarly, program impacts for 
those programs that conducted random assignment before MTO or within its first few days were similar to the 
impacts for programs that conducted random assignment later (not shown). 


12 


deviate from it if the evaluation team deemed it necessary.*! On average, 69 percent of eligible 
young people were randomly assigned to the YouthBuild group and 31 percent were assigned to 
the control group. Only nine programs (representing 16 percent of study participants) used a ratio 
above 80:20. 


Data and Methods 


The analysis presented in this report is based on several data sources. First, three surveys were 
administered to a random subset of study participants, at 12, 30, and 48 months after they entered 
the study. The surveys obtained information on participation in education and training, educa- 
tional attainment, work, family formation, youth development, civic engagement, involvement 
with the criminal justice system, and child support. The first two waves of surveys achieved 80 
percent response rates, with very small differences between the program and control groups. The 
48-month survey achieved a final response rate of 78 percent, with a small difference in response 
rates (of 4 percentage points) between the program and control groups. A total of 2,721 partici- 
pants provided responses to the 48-month survey.” 


Administrative records on employment and earnings were obtained for the full sample 
from the National Directory of New Hires, which contains quarterly wage data for workers in 
employment covered by the unemployment insurance (UI) system. These data miss employment 
that is not covered by the UI system, including informal work and self-employment. Data on 
postsecondary enrollment were obtained for the full sample from the National Student Clearing- 
house, which tracks enrollment and degree receipt nationally. Although it captures over 90 per- 
cent of postsecondary enrollment in the United States, its coverage varies for different types of 
institutions. Its coverage is highest (estimated to be 99 percent) for four-year public institutions, 
for example, and lowest (estimated at 48 percent) for for-profit institutions.** Thus, both types of 
records are important complements to the surveys, but they may miss some types of employment 
and postsecondary enrollment. The advantage of these data, however, are that they represent an 
independent source of outcome information, as they are not subject to problems of recall or a 
desire to self-report more positive outcomes than have actually occurred. 


A grantee survey was administered in fall 2012 to all 110 YouthBuild programs funded 
by DOL or CNCS in 2011, which includes those programs selected for the study. The survey 
asked administrators to provide information about their programs’ years in operation, funders, 


31Fifteen programs met the 60:40 ratio; 30 programs had ratios ranging between 61:39 and 70:30; 18 pro- 
grams had ratios between 71:29 and 80:20; and the remaining 9 programs had ratios between 81:19 and 89:11. 


32 Appendix B includes an analysis of survey response. 
33Dynarski, Hemelt, and Hyman (2015). 
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operating budgets, staff structures and staff experience levels, construction work site characteris- 
tics, recent recruitment and enrollment experiences, stipends, and program component character- 
istics. These data are used to compare the study programs with the larger population of programs 
from which they were drawn. 


Additionally, all programs receiving DOL funding are required to enter data on partici- 
pant characteristics, service receipt, and outcomes into the DOL YouthBuild management infor- 
mation system, a database designed to record program operations and guide program manage- 
ment. The research team used these data to assess the reliability of the findings on YouthBuild 
participation shown in the surveys. All programs receiving CNCS funding entered quarterly data 
into a separate management information system. While these data do not provide detailed infor- 
mation on participation, they do indicate formal enrollment status. 


Finally, detailed data on costs were collected either on site or through phone calls made 
to fiscal supervisors at each of the participating sites. These data included overall costs, as well 
as breakdowns in cost by grantee personnel, administrative costs, and participant costs, including 
stipends. Cost data from comparable programs were used to assess the relative cost of YouthBuild 
services. Chapter 5 details the results of these cost calculations and comparisons. 


In several of the chapters of the report, results from this evaluation are compared with 
those from a number of other evaluations of programs that target similar young people, including 
Job Corps, National Guard Youth ChalleNGe, Youth Corps, and Jobstart. These comparisons are 
made to provide context for the impact and cost findings for YouthBuild. When making the com- 
parisons, it should be kept in mind that two of the programs are residential and one of the studies 
was conducted more than 20 years ago. A fuller description of these four and selected other youth 
programs and their results is provided in Appendix Table A.1. 


Because young people in this evaluation were randomly assigned either to the program 
group or to the control group, the effects of the program can be estimated as the differences be- 
tween the two groups’ outcomes. (Appendix Table B.1 presents a comparison of the characteris- 
tics of the program and control groups, showing that the two groups were similar on average when 
they enrolled in the study.) These differences between the full program and control groups (often 
referred to as “intent-to-treat” effects) are the main focus of this report and represent the effect of 
the program on the average outcomes of young people assigned to the program group, whether 
or not they participated in YouthBuild. Occasionally, the report will mention “impacts per partic- 
ipant” (sometimes referred to in other research as “treatment-on-the-treated” effects), which rep- 
resent the effects of the program on those young people in the program group who actually par- 
ticipated in YouthBuild. Effects per participant are estimated by dividing the effects on the full 
program group by the fraction of the program group who participated in YouthBuild. 
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Impacts are estimated for each outcome using regression models, in which the outcome 
of interest is regressed on an indicator for program status and several variables measured at or 
before the time of random assignment. These additional variables improve the precision of the 
impact estimates; they include the participant’s age, gender, educational level, parent status, and 
race or ethnicity. Variables are also included for each individual program in order to account for 
variation in the random assignment ratio among participating programs. See Appendix B for more 
detail. 


Finally, as the number of outcomes that are examined increases, the probability of ob- 
taining impacts that are statistically significant simply by chance also increases. Although the 
estimates in this report are not formally adjusted to account for multiple hypothesis testing, the 
analysis does attempt to address this risk by limiting the number of outcomes examined. In addi- 
tion, effects that do not appear to be part of a larger pattern are given less weight in the discussion. 


Findings Presented in Earlier Reports 


As noted above, two earlier reports have been released as part of this evaluation: an implementa- 
tion report and an interim impact report. The findings presented in each of these reports have 
implications for the findings presented in this report, and thus are summarized below. 


The implementation report described the implementation and operations of the program 
across all grantees participating in the evaluation. Overall, this report found that the program was 
implemented well across the participating grantees, and generally had high fidelity to the Youth- 
Build model, likely driven by the centralized direction provided through the law, DOL’s regula- 
tions, and the guidance provided by YouthBuild USA, though the program also allowed for some 
variation according to program and community contexts. Fidelity was consistent and highest in 
the vocational services component, and varied somewhat more in the leadership development and 
postsecondary educational components.* Programs also varied substantially in the length of their 
MTO, and the extent to which they used this component as a means for screening young people 
before program enrollment. Given these findings, Chapter 4 of this report presents an analysis of 
the extent to which fidelity to the postsecondary education and career development components, 
as well the length of MTO, are associated with program impacts. The implementation report noted 
additional factors that varied substantially across grantees, including whether the grantee man- 
aged the construction site in house, whether they offered GED preparation only (or also offered 
assistance toward a high school diploma), the number of years the program had been operating, 
and whether they required a minimum educational or math skills level to enroll in the program. 


34One part of the leadership fidelity rating was the presence of a Youth Policy Council. Programs must have 
an active Youth Policy Council in order to become members of YouthBuild USA’s Affiliated Network. How- 
ever, it is not an element that DOL requires of its grantees. 
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The association between these factors and impacts on key outcomes is also considered in Chapter 
A of this report. 


The interim impact report presented impact findings for the two and a half years follow- 
ing participants’ enrollment into the study. Very similar to the present report, the interim report 
relied on administrative data on education, employment, earnings, as well as surveys of partici- 
pants. The interim report identified a number of key findings, including the following: 


e Approximately 75 percent of those assigned to the program group actually 
participated in YouthBuild. Approximately half of these participants reported 
that they graduated from the program within 12 months. 


e Overall, participants rated their experiences within the YouthBuild program 
favorably. In particular, participants valued most highly the construction or 
other job training, leadership development training, and the counseling they 
received in the program. 


e Yong people invited to enroll in the program had higher rates of participation 
in education, training, and personal development activities than those who 
were assigned to the control group. This was in spite of the fact that control 
group members also had relatively high rates of participation in education and 
training activities (presumably through alternative sources in their communi- 
ties). 


e Program group members had higher rates of high school equivalency creden- 
tial (or, GED credential) attainment, college enrollment, and participation in 
vocational training. 


e YouthBuild led to a small increase in employment rates during the second year 
after participants’ entry into the study, and a small increase in self-reported 
wages and earnings 30 months after entry into the study. 


e Program group participants had higher rates of civic engagement than control 
group members, particularly in volunteering (Something especially empha- 
sized by programs receiving CNCS funding), but both groups were compara- 
ble on other measures of youth development or attitudes. 


e There were no apparent effects of the program on participants’ involvement 
with the criminal justice system. 
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Each of these findings occurred within the two and a half years following participants’ 
enrollment into the study. The present report assesses whether the impacts observed during that 
earlier period were maintained (or enhanced) after four years. 


Organization of This Report 


The remainder of the report presents findings on participation in YouthBuild (among young peo- 
ple assigned to the program group) and program effects. Chapter 2 summarizes the findings from 
the process study on program implementation, presents data on participation in YouthBuild, and 
describes impacts on service receipt over the four-year follow-up period. Chapter 3 presents the 
program’s effects on a range of outcomes, including educational attainment, work and earnings, 
measures of youth development, and involvement in the criminal justice system. Chapter 4 pre- 
sents effects on important outcomes for selected subgroups, including subgroups based on partic- 
ipant characteristics and varying program features. Chapter 5 presents a detailed cost analysis, as 
well as a limited cost-benefit analysis. Chapter 6 provides a summary of the report and of the 
overall evaluation. 


17 


THIS PAGE INTENTIONALLY LEFT BLANK 


Chapter 2 


Implementation, Participation, and Service Receipt 


This chapter discusses program implementation findings, participating young people’s experi- 
ences with the YouthBuild program, and young people’s participation in YouthBuild and other 
services. The first section reviews findings from the earlier implementation report. The second 
section reviews and updates the interim impact report’s findings on young people’s participation 
in YouthBuild. The third section presents impacts on participation in education, training, and 
other services over the 48-month follow-up period. 


Implementation Findings 


In 2015, the first report of this evaluation, Adapting to Local Context, was released, presenting 
implementation findings for the 75 participating YouthBuild programs. Extensive implementa- 
tion data were collected and analyzed for that report, including multiday visits to each partici- 
pating program and an online survey of programs. The participating programs were diverse and 
were representative of the YouthBuild programs operating nationwide at the time. They varied 
in their geographic locations, the lengths of time they had been in their communities, their 
organizational structures, and their funding and staffing arrangements. Overall, the evaluation 
found that programs were implementing the YouthBuild model with fidelity. 


It is worth noting, however, that the YouthBuild program model described in Chapter 1 
is not highly prescriptive. Instead, it is designed to allow variation based on the program and 
community contexts. For example, there was considerable variation in the format of the educa- 
tional services that different YouthBuild programs offered. Yet even though the content and 
format of classes varied from program to program, most shared the goal of providing partici- 
pants individually tailored instruction and academic support. Small class sizes helped to 
promote positive relationships between instructors and participants. For vocational training, 
about one-fifth of programs in the evaluation offered training in fields other than construction, 
such as health care, culinary arts, and computer technology. Training staff members reported 
often being stretched thin, and staff members overall reported that the economic downturn had 
significantly affected programs’ ability to implement construction-related vocational training. 
Specifically, it limited programs’ ability to provide quality construction experiences to partici- 
pants and limited the marketability of the job skills programs taught. Programs grappled with 
ways to diversify their operations through new partnerships and to provide new training 
opportunities for participants. 
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Programs embraced the culture and value system of YouthBuild to varying degrees. 
This variation appeared most often in the leadership development component, a defining feature 
of YouthBuild, as noted in Chapter 1. While fidelity to the YouthBuild program model was 
high overall, not all programs were equally faithful in implementing the leadership development 
component. For example, one-fourth of the participating programs did not have a functioning 
Youth Policy Council, a committee of young people that plays an active role in setting decisions 
and policies that affect the program.' Programs that did not implement the leadership develop- 
ment component as faithfully as others often had fewer resources and less ability to dedicate 
staff time to these activities. 


Many programs adopted flexible staffing arrangements to deliver supportive, transition- 
al, and follow-up services. Programs often needed to distribute the task of delivering these 
services across multiple staff members. For example, fewer than half of the programs had a 
designated job developer to assist participants with job searches and job placement. Programs 
also often combined life skills and work-readiness training into one class, diluting their intensity 
but meeting the requirement to offer these services. 


YouthBuild programs typically were not alone in providing services to young adults in 
the communities where they operated. Although all the communities in this evaluation had other 
organizations offering some of the same services as YouthBuild, these alternative services 
rarely matched the breadth and depth of those provided by YouthBuild. The services were 
usually not all available through the same provider. Therefore, it was possible for a young 
person to gain access to the same service components as provided by YouthBuild, but the 
individual would need to visit multiple providers to do so, and it seems unlikely that those 
providers would have coordinated their services in a way that would mimic the YouthBuild 
experience. Similarly, few alternative programs in these communities seemed to create empow- 
ering environments for their participants that could rival the environments developed by most of 
the YouthBuild programs in this study. See Box 2.1 for more information on the availability and 
receipt of alternative services in YouthBuild sites. 


Participation in YouthBuild 


The second report of this evaluation, Building a Future, presents impact findings for the 30- 
month follow-up period. This section reviews program participation and experience findings 
from that report and updates program participation findings. 


‘Programs must have an active Youth Policy Council in order to become members of YouthBuild USA’s 
Affiliated Network. However, it is not an element that the U.S. Department of Labor requires of its grantees. 
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Box 2.1 


Service Contrast Among YouthBuild Sites 


YouthBuild provided a holistic suite of services that was more intensive than those available at 
most alternative service providers in the local communities. However, all YouthBuild commu- 
nities had alternative services available to control group members, as shown in the chart below. 
Additionally, 14 communities in the evaluation were home to multiple YouthBuild programs, 
4 communities had alternative service providers offering a range of services similar to that of 
YouthBuild, and 14 communities had a Job Corps program — a residential program offering 
similar services to a similar demographic as YouthBuild — in the same county. 


According to administrative program data, less than 1 percent of control group members 
attended YouthBuild programs involved in the evaluation, but it is not possible to know 
whether any control group members participated in YouthBuild programs that were not a part 
of the evaluation. Survey data show that, although control group members sought out services 
at high rates during the study period, the program group received substantially more services in 
all domains. 


Communities in the Study with 
Alternative Services (%) 


Education services 
Work-readiness services 
Supportive services 


Case management 


Workforce training 


Young people were asked about their participation in YouthBuild in the three follow-up 
surveys. Most participants had left the program before the 12-month survey was administered. 
At 12 months, 74 percent of the program group reported that they had received YouthBuild 
services or participated in YouthBuild activities in the past, and participation rose to 80 percent 
by 30 months. As shown in Appendix Table C.1, participation rose again to 82 percent by 48 
months. These figures indicate that some program group members may have returned to 
YouthBuild after leaving the program during their initial enrollment. 
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Program participation was broadly defined and does not necessarily indicate that a 
young person formally enrolled in YouthBuild. As discussed in Chapter 1, in many cases 
random assignment occurred after Mental Toughness Orientation (MTO), so a survey respond- 
ent’s report of “participating” in YouthBuild might only mean that the individual had attended 
some part of MTO. However, for the interim report, these survey responses were compared in 
multiple locations with the programs’ administrative data on participation, and participation 
rates were similar across the two data sources.’ This similarity suggests that a significant portion 
of the program group did participate in YouthBuild beyond MTO. 


As reported in the interim report, at 12 months, among the 74 percent of program group 
members who reported receiving services, young people remained in YouthBuild for an average 
of eight months, and 50 percent reported completing the program. Once involved in program 
activities, most young people (87 percent) participated for more than three months. Again, these 
findings indicate that young people participated in YouthBuild beyond MTO and that the 
program staff members were able to engage them in the program model.* 


Young people in the program group described a variety of reasons for either never at- 
tending YouthBuild or not completing it. The most common reason for not participating in the 
program was transportation issues (32 percent of young people reported this reason), an obstacle 
program staff frequently cited in the implementation report as a barrier to attendance. Other 
common reasons for not participating included finding another job, family or health issues, and 
a dislike of the program (the schedule, staff, or other participants). Notably, 10 percent reported 
not participating in YouthBuild because they had become incarcerated. 


Overall, young people who attended YouthBuild rated the services they received and 
the program’s staff highly. Highly rated program services included the general program experi- 
ence, the construction or job-training component, the counseling, and the leadership training. 
Young people seemed slightly less satisfied with the services they received after they left the 
program — such as assistance finding a job or other forms of help — and with the staff mem- 
bers who delivered them. Some young people were in contact with the program relatively 


The timing of enrollment — and the definition of the term — varied among the 75 participating pro- 
grams, so it was not possible to ask in the survey interviews about formal enrollment into core YouthBuild 
services. 

3Survey responses were compared with administrative data collected in 69 programs, representing 87 per- 
cent of the young people in the sample. 

‘These participation levels are similar to those found in other evaluations of youth programs. Job Corps’ 
program participation rate was 73 percent, and young people reported staying in the program for an average of 
eight months (Schochet, Burghardt, and Glazerman, 2001). In the National Guard Youth ChalleNGe evalua- 
tion, 83 percent of the program group “registered” and began the residential pre-ChalleNGe orientation phase 
(Bloom, Gardenhire-Crooks, and Mandsager, 2009). 
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frequently after they left, while others were in contact relatively infrequently. As reported in the 
interim impact report, at 30 months, one-fourth of participants said they were in contact at least 
once per month, one-fourth reported contact several times per year, and the remaining half were 
in contact once per year or not at all. Note that when young people were asked this question at 
30 months, most had been out of YouthBuild for almost two years, which is well beyond the 
12-month period for which programs receive funding to provide post-program follow-up 
services. 


These participant ratings echo the findings about transitional and post-program services 
presented in the implementation report. They also echo an earlier study that surveyed Youth- 
Build graduates and found that many of them wanted more contact with the program after 
graduation.® Most programs did not have staff members dedicated to cultivating employers, 
identifying job openings, and placing people in jobs. Smaller programs especially felt con- 
strained by funding and were unable to devote enough staff time to support young people with 
job searches and job placement. Staff members said it was sometimes challenging to follow up 
with participants after they left the program because their living situations and contact infor- 
mation changed frequently. Other challenges to follow-up included a lack of staff time and 
difficulty getting alumni to come to YouthBuild offices. 


Young people enrolled in the study and assigned to the control group were prohibited 
from enrolling in the participating YouthBuild program for two years. As noted earlier, these 
sample members were given information about other appropriate services in the community. An 
analysis of administrative program data shows that less than 1 percent of control group mem- 
bers formally enrolled in participating YouthBuild programs following random assignment (not 
shown). Note that this program information was only available for 69 programs, representing 87 
percent of the young people in the study sample. In addition, at the time of the evaluation, there 
were 14 cities that were home to multiple YouthBuild programs, including programs not 
participating in the evaluation. Unfortunately, it is not possible to measure control group 
participation at YouthBuild programs not involved in the evaluation. Some members of the 
control group did report in the survey interviews that they participated in YouthBuild. However, 
the question was asked in a very general way and could include a young person’s participation 
in the application process and MTO, in some cases. 


Impacts on Service Receipt Through 48 Months 


The integration of educational, vocational, and supportive services in the YouthBuild model 
suggests that young people in the program group should receive more and a wider range of 


Levine (2012). 
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services than their control group counterparts. As discussed in Chapter 1, these services — 
combined with the contextual factors of the program, the participants, and the community — 
can lead to positive outcomes for participating young people. However, young people who 
made it through the lengthy screening process were a motivated group at the time they applied. 
Therefore, the young people assigned to the control group are likely to have sought out alterna- 
tive services on their own. This section examines the differences between the two groups in 
participation in educational, vocational, and other services. 


One year after random assignment, both research groups had participated in many ser- 
vices, but program group members had participated at statistically significantly higher levels in 
all types of services. These impacts held through 48 months, but decreased in magnitude as 
most program group members had finished participating in YouthBuild by 12 months. During 
the post-program period, control group service receipt increased at a faster rate than program 
group service receipt. 


The vast majority of the program group said that they participated in services at Youth- 
Build, and the control group reported participating at a variety of locations. These findings 
reinforce those of the process study: Alternative service providers in communities in the study 
rarely offered the same breadth and depth of services as YouthBuild. However, as noted, 
YouthBuild’s screening process led to a highly motivated sample, so it is not surprising that 
service receipt rates for the control group were high and also increased over time. 


Services were broken up into three domains: education, job training, and personal development. 
During the 48-month follow-up period, the young people in both study groups reported receiv- 
ing education-related services more often than services in other domains, followed by job- and 
training-related services. Figure 2.1 shows total service receipt by domain at 12 and 48 months, 
along with the increases in service receipt between survey waves. ° The leftmost column shows 
participation in services in the 12 months following random assignment (that is, the program 
period). Impacts on service receipt were largest during this period across domains.’ The middle 
columns show the increase in services in the 12- to 30-month follow-up period and the 30- to 
48-month follow-up period, which, for most program group members, corresponded to the post- 
program period. Service receipt levels increased in both groups in both follow-up periods, but 
the rate of new participation among control group members was around double the rate among 


°Figure 2.1 is restricted to the sample that responded to all three surveys in order to more accurately repre- 
sent service receipt growth over time; however, results are similar for the full sample. For impacts on service 
receipt for the full sample, see Appendix Table C.1. 

"Impacts on 12-month service receipt for the sample of young people who responded to all three surveys 
were 18.3 percentage points for educational services, 34.6 percentage points for job training services, and 29.2 
percentage points for personal development services. 
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Figure 2.1 


Participation in Services Throughout the Study Period 
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Figure 2.1 (continued) 
SOURCES: MDRC calculations based on responses to the 12-, 30-, and 48-month follow-up surveys. 


NOTES: Sample includes participants who responded to all three surveys (sample size = 2,225). 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


program group members across domains. In other words, a larger proportion of control group 
members without prior exposure to services reported seeking services in the post-program 
period compared with program group members. The last column shows total service receipt 
during the 48-month follow-up period. Though the gaps between program and control group 
participation had narrowed between 12 and 48 months, the program group still received 
statistically significantly more services across domains. 


As shown in Appendix Table C.1, control group participation failed to surpass that of 
the program group in any individual service category at 48 months. High school equivalency 
preparation was the service control group members participated in most and the service in 
which they gained the most ground during the post-program period. Still, at 48 months there 
was a 7 percentage point impact on high school equivalency preparation, which declined from 
an 18 percentage point impact at 12 months. This pattern, and the broader pattern of educational 
service receipt over time, is unsurprising given (1) the implementation study finding that the 
majority of study participants applied to YouthBuild primarily because it provided an oppor- 
tunity to earn a General Educational Development (GED) credential, and (2) that about 50 
percent of low-income young people who drop out of high school go on to earn a high school 
equivalency credential, as noted in Chapter 1. Thus, it stands to reason that participants in both 
groups were highly motivated at the time of study enrollment to seek out educational services, 
and GED services in particular. Control group members also gained significant ground in on- 
the-job training and job search assistance, with impacts decreasing by 10 and 8 percentage 
points, respectively, between the 12- and 48-month surveys. 


In conclusion, YouthBuild provides services that it would take eligible young people 
years to experience on their own, and even after four years the control group had yet to close the 
service gap in any domain. The control group pursued services at a high rate throughout the 
follow-up period, but there were still relatively large and statistically significant differences 
between program and control group participation after four years. These findings corroborate 
the implementation study finding that YouthBuild provides a uniquely broad and intensive 
service environment that eligible applicants — despite being a highly motivated group that 
values the services YouthBuild offers — would struggle to match through a combination of 
service providers in their communities. 
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Chapter 3 


Impacts Through Four Years 


As shown in the previous chapter, young people in the study continued to participate in educa- 
tional and training activities throughout the four-year period. Although the differences had nar- 
rowed by the 48-month point, young people in the program group were still more likely to have 
participated in educational, training, and personal development services than their control group 
counterparts. In addition, the earlier impact report documented that by the 30-month point, young 
people in the program group were more likely to have obtained a high school equivalency cre- 
dential and have been enrolled in postsecondary education than those in the control group. 


Both of these factors suggest that YouthBuild may lead to continued effects on education, 
training, work, and other outcomes. This chapter presents the program’s effects through four years 
on education, employment, youth development, involvement in the criminal justice system and 
other outcomes. Primary outcomes, or those that should be most directly affected by participation 
in YouthBuild, include educational attainment, employment, and positive youth development. 
Secondary outcomes, whose effects should arise through effects on the primary outcomes, are 
family formation, living arrangements, and involvement in the criminal justice system. 


The data used for the analyses are administrative records on employment, earnings, and 
college enrollment, and a 48-month survey. Effects are estimated using all program and control 
group study participants, even though some young people in the program group never partici- 
pated in YouthBuild. For this reason, effects are sometimes discussed in terms of “impacts per 
participant.” Impacts per participant for selected outcomes are presented in Appendix Table B.2. 


In sum, the program led to continued positive impacts on high school equivalency receipt 
and college enrollment over the full 48-month period. Effects on employment and earnings were 
also positive, but only apparent in work reported on the survey and not in work reported to the 
unemployment insurance (UJ) system. Potential reasons for this difference are discussed below. 
There were few significant impacts on measures related to youth development or involvement in 
the criminal justice system. 


Impacts on Education 


Educational services are a key component of the YouthBuild model and include classes to help 
students complete their high school diplomas or their high school equivalency credentials, voca- 
tional services to prepare young people for careers, and services that help put students on a path 
to postsecondary education. The interim report found that, through month 30, the program led to 
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an increase in high school equivalency credential receipt of 14 percentage points, an increase in 
vocational school enrollment of 10 percentage points, and an increase in college enrollment of 
about 6 percentage points. 


Table 3.1 presents impacts on education and training through Year 4. The data are from 
the 48-month survey and the National Student Clearinghouse (NSC) records on postsecondary 
enrollment. All outcomes in this table reflect activity reported during the full 48-month follow- 
up period. Data from the survey (top panel) show that the program group was still more likely 
than the control group to have obtained a high school equivalency credential during the period — 
34.5 percent of the program had this credential, compared with 23.5 percent of the control group, 
for an impact of 11 percentage points. The effect is only slightly smaller than that observed at 30 
months (14 percentage points). The effect per participant on credential receipt through month 48 
is 14.8 percentage points. Consistent with findings presented in the interim report, the program 
did not lead to a difference in high school diploma receipt.! 


Young people in the program group were also more likely to have enrolled in vocational 
school within 48 months and to have received a trade license or certificate. To measure vocational 
school enrollment, the survey asked respondents about formal enrollment in technical, business, 
or trade school, suggesting that this enrollment is probably not through YouthBuild. For example, 
32.9 percent of the program reported enrollment in a vocational school, compared with 21.7 per- 
cent of the control group, for a difference of 11.2 percentage points. This impact on vocational 
school enrollment is very similar to that found at the 30-month point. The rates of participation 
for both groups are also very similar to those reported at 30 months, suggesting that there was not 
much new participation in vocational schools after that point. 


Two points of comparison are provided by the Job Corps and National Guard Youth 
ChalleNGe findings, also based on randomized controlled trials. There are a few issues to keep 
in mind, however, when making these comparisons. Both Job Corps and ChalleNGe are residen- 
tial programs, meaning that they are typically more intensive and they remove young people from 
sometimes disruptive home and community environments. In addition, the findings from Job 
Corps are based on a study sample that enrolled in the program in the mid-1990s, a very different 
context from today. Nonetheless, as two often-cited programs for young people, it is worth com- 
paring them with YouthBuild to provide context. 


‘As noted in Chapter 1, a fair number of YouthBuild programs are high school diploma granting schools. 
Separate analyses indicated that among this subgroup of programs in the evaluation, YouthBuild did increase 
high school diploma receipt. 
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Table 3.1 


Impacts on Education and Training at 48 Months 


YouthBuild Control Difference 

Outcome (%) Group Group (Impact) P-Value 

Outcomes based on survey responses 

Has high school diploma or equivalency credential 54.6 45.7 8.9*** 0.000 
High school diploma 20.1 22.1 -2.0 0.155 
High school equivalency credential 34.5 23.5 11.0*** 0.000 

Enrolled in vocational school since random assignment® 32.9 21.7 11.2*** 0.000 

Received school-based trade license/training certificate 
since random assignment? 5.3 3.4 1.9* 0.058 

Enrolled in postsecondary courses since random assignment 26.9 21.7 5.3*** 0.004 
4-year college or university 5.4 3.4 2.0** 0.032 
2-year or community college 25.1 20.1 5.0*** 0.006 

Received postsecondary degree since random assignment 1.8 1.3 0.4 0.419 
Associate’s degree 1.3 0.9 0.4 0.400 
Bachelor's degree 0.2 0.1 0.1 0.482 
Other degree 0.5 0.8 -0.3 0.397 

Sample size (total = 2,721 1,784 937 

Outcomes based on administrative enrollment data 

(since random assignment) 

Attended college 21.3 12.7 8.6*** 0.000 
Enrolled in a 4-year institution 45 3.5 1.0 0.137 
Enrolled in a 2-year institution 17.7 9.9 7.8*** 0.000 
Enrolled in a less-than-2-year institution 0.2 0.0 0.2 0.199 
Full time 9.5 6.7 2.8*** 0.004 
Part time 17.1 8.7 8.4*** 0.000 

Received a degree 1.7 0.9 0.8** 0.049 
Certificate 1.2 0.6 0.6* 0.075 
Associate’s 0.2 0.0 0.2* 0.084 
Bachelor's 0.1 0.2 -0.1 0.389 
Master’s 0.1 0.0 0.1 0.450 

Sample size (total = 3,929) 2,700 1,229 
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(continued) 


Table 3.1 (continued) 


SOURCES: MDRC calculations using data from the National Student Clearinghouse and responses to the 
48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


“Vocational school includes technical, business, and trade schools. 
Trade license/training certificate includes technical, business, and trade certificates. 


YouthBuild’s impacts on high school equivalency credential receipt are somewhat 
smaller than those found in Job Corps and National Guard Youth ChalleNGe. After four years, 
42 percent of the program group in the Job Corps evaluation and 27 percent of the control group 
had earned high school equivalency credentials, for an impact of 15 percentage points.? In Chal- 
leNGe, the impact after three years was 22 percentage points.° 


The next set of rows in Table 3.1 presents effects on postsecondary enrollment. Given 
the positive effects on high school equivalency credential receipt and the emphasis many Youth- 
Build programs place on postsecondary preparation, it is reasonable to expect effects on college 
enrollment.‘ In addition, as noted earlier, positive effects on college enrollment were found at the 
30-month point. The data show that the positive effects persisted through Year 4. 


About 27 percent of young people in the program group reported having enrolled in post- 
secondary classes since random assignment, about 5 percentage points more than the control 
group. The effect per participant is somewhat larger, at about 7 percentage points. (See Appendix 
Table B.2.) The survey data show that the largest effect on college enrollment was in two-year 
colleges. In addition, by Year 4, very few enrollees had earned a degree (less than 2 percent). It 
is unlikely that many young people would have earned a four-year degree by the time of the 48- 
month survey. Rates of associate’s degree receipt are also expected to be low, given that just over 
half of the young people entered the study having completed tenth grade or less. 


NSC data on college enrollment are also shown in the bottom panel of Table 3.1. These 
records capture student enrollment at most postsecondary institutions in the United States, al- 
though they cover notably fewer for-profit institutions such as many technical schools. Because 
the NSC captures a fairly broad set of data, there may be some overlap with enrollment as meas- 
ured by these data and enrollment in vocational school reported on the survey. 


*Schochet, Burghardt, and McConnell (2006). 
3Millenky, Bloom, Muller-Ravett, and Broadus (2011). 


4Nearly all programs in the study provided some form of postsecondary educational preparation, although 
some programs offered more intensive services than others. See Wiegand et al. (2015) for more information. 
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The NSC data show a similar impact on college enrollment as does the survey, although 
lower rates of enrollment. For example, about 25 percent of young people in the program group 
reported on the survey that they had enrolled in a two-year college since study entry, compared 
with 18 percent reported in the NSC data. This difference may reflect the fewer data the NSC 
captures for technical schools. The NSC data also show low rates of degree or certificate receipt 
and very small impacts on certificate and associate’s degree receipt, of 0.6 percentage points and 
0.2 percentage points, respectively. 


To put these rates and effects in context, the four-year findings from Job Corps also show 
very small rates of college degree receipt (less than 2 percent for both research groups) and no 
impact on this outcome. However, Job Corps did lead to much larger effects on vocational certif- 
icate receipt, of about 22 percentage points. ChalleNGe, on the other hand, had no effect on cer- 
tificate receipt (although vocational training was not part of its core services), but did lead to 
moderate increases in postsecondary enrollment. The program increased the percentage of young 
people who reported earning some college credit over the three-year period (an increase of 16 
percentage points). By the time of the three-year survey, however, many fewer young people 
reported being enrolled in college, and the impact on enrollment had fallen to 4 percentage points. 
ChalleNGe similarly led to a very small increase in associate’s degree receipt, of less than 1 per- 
centage point. 


The low rates of degree receipt for ChalleNGe and YouthBuild highlight the problem of 
low persistence among young people who attend college. Figure 3.1 presents enrollment rates for 
the YouthBuild sample by follow-up quarter using the NSC data. The data show enrollment rates 
for the program group peaked at 10 percent in the fourth quarter after study enrollment. This 
finding is consistent with the time young people spent in YouthBuild and their transition out of 
the program and into college. However, enrollment rates fall back to less than 6 percent by the 
seventh quarter. Although the program group’s enrollment rates remain slightly above the control 
groups for the remainder of the follow-up period, few of the differences are statistically signifi- 
cant. 


Although the low rate of persistence in college is discouraging, it is consistent with find- 
ings from other studies of similar groups. Despite an increase in the past decades in access to and 
enrollment in college among low-income individuals, persistence and completion rates remain 
low, particularly among those who enroll in community colleges. Only about a third of first-time 
community college students complete a degree within six years.° Reasons for the lack of persis- 
tence range from financial barriers, academic challenges, and the difficulties of balancing school 


°Ma and Baum (2016); Snyder and Dillow (2013). 
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Figure 3.1 


College Enrollment by Quarter 
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SOURCE: Calculations based on data from the National Student Clearinghouse. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 
Sample size = 3,929. 


and family or work responsibilities. There has been increased focus in recent years on policies 
designed to help low-income students persist in college, once they have enrolled, and a number 
of recent evaluations have identified promising strategies.® The data in Table 3.1 suggest that the 
young adults in the YouthBuild study could benefit from some type of ongoing assistance while 
in college. 


The City University of New York’s Accelerated Study in Associate Programs (ASAP), for example, almost 
doubled graduation rates after three years among low-income students who needed developmental classes 
(Scrivener et al., 2015). Effects after six years were smaller, but still notable (Gupta, 2017). Similarly, the Open- 
ing Doors Demonstration tested a variety of strategies to increase persistence and completion at six community 
colleges around the nation and identified several promising strategies (Scrivener and Coghlan, 2011). 
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Impacts on Employment and Earnings 


Overall, YouthBuild’s effects on work and earnings were small at the 30-month point. The pro- 
gram led to an increase in employment for the less-educated subgroup, but not for the sample as 
a whole, although there was a small, positive effect on wages earned. Trends in employment over 
time, using the administrative records data, suggest that impacts might have increased as young 
people made the transition out of YouthBuild and college and into work. The positive effects on 
high school equivalency receipt and postsecondary enrollment at the 30-month point also point 
to potential employment impacts by Year 4. When interpreting the effects on work and earnings, 
it is important to remember that the study participants were on average 24 years of age after four 
years of follow-up. Other research has found that young peoples’ work and earnings patterns do 
not tend to settle down until they are in their late 20s.’ In other words, young people’s earnings 
in their early 20s are not strongly correlated with, and may not present an accurate picture of, 
longer-term career earnings. 


Table 3.2 presents impacts on employment and earnings for the 48-month follow-up pe- 
riod using survey data and administrative records data from the National Directory of New Hires 
(NDNH), which provides quarterly wage records covered by the UI system. As shown in the top 
panel, and based on the survey data, 85 percent of young people in the control group reported 
working at some point during the follow-up period, highlighting that the study sample is a moti- 
vated group of young adults. Nevertheless, YouthBuild led to an increase in employment of 4.1 
percentage points during this period. Only about half (46.4 percent) of young people in the control 
group, however, were working at the time of the survey. The program led to an increase in current 
employment of 4.5 percentage points. 


Data on wages show that young people who worked were earning more over time. At the 
30-month point, only a third of workers in the control group were earning more than $10 per hour. 
By the 48-month point, nearly half of workers earned that amount (that is, 23 percent earning 
more than $10 per hour divided by 46.4 percent currently working). The program also led to a 
small increase in the number of young people earning more than $10 per hour, an effect of 4.6 
percentage points that is similar in size to the effect found at 30 months.* As a result, young people 
in the program group earned on average $33 more per week than those in the control group, an 
increase of 19 percent. 


7Chetty, Hendren, Kline, and Saez (2014); Haider and Solon (2006). 


8See Appendix Table C.2 for impacts on other job characteristics. The program overall did not have notable 
effects on where participants worked, in terms of industry, although it did lead to a small increase, of 3.3 per- 
centage points, in employment in jobs that offered health insurance. 
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Table 3.2 


Impacts on Employment and Earnings at 48 Months 


YouthBuild Control Difference 
Outcome Group Group (Impact) P-Value 
Outcomes based on survey responses 
Ever employed since random assignment (%) 89.4 85.3 4.1*** 0.003 
Currently employed (%) 50.9 46.4 4.5** 0.033 
Works full time (35+ hours/week) 32.5 29.4 3.0 0.122 
Self-employed 10.0 9.7 0.3 0.830 
Works through a temp agency 7.8 7.4 0.5 0.691 
Earning $10/hour or more 27.5 23.0 4.6** 0.014 
Current average weekly earnings ($) 206.7 174.1 32.6*** 0.005 
Sample size (total = 2,721) 1,784 937 
Outcomes based on New Hires data 
Employment since random assignment (%) 
Employed in Year 1 48.9 51.3 -2.4 0.120 
Employed in Year 2 62.9 59.7 3.2** 0.049 
Employed in Year 3 68.4 66.3 2.1 0.185 
Employed in Year 4 71.6 73.3 -1.7 0.278 
Earnings since random assignment ($) 
Earnings in Year 1 2,093 2,408 -314** 0.016 
Earnings in Year 2 3,735 3,937 -201 0.293 
Earnings in Year 3 5,442 5,331 111 0.652 
Earnings in Year 4 6,980 6,729 251 0.407 
Sample size (total = 3,878) 2,662 1,216 


SOURCES: MDRC calculations from the National Directory of New Hires (NDNH) and responses to the 


48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 


site fixed effects. 


Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Social Security numbers are unavailable for some sample members, who therefore could not be 


matched to the NDNH database. 
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The bottom panel of the table presents data on employment and earnings from the NDNH 
records. Employment rates for the YouthBuild sample increased fairly rapidly over time, con- 
sistent with this group’s aging into young adulthood. (Quarterly data are presented in Figure 3.2.) 
However, the sample’s employment rates were relatively high for this population. Nationally, 
among 20- to 24-year-olds, 64.6 percent worked in 2016.° 


The NDNH records data match the survey data relatively well on comparable measures. 
About 87 percent of the full sample was employed at some point based on the records data, and 
a little over 50 percent were employed in the final follow-up quarter (not shown). These rates 
roughly match rates of ever working and currently working from the survey. 


However, in terms of impacts, the records data do not match the survey in showing an 
increase in work or earnings for the program group, with the exception of a small increase in 
employment in Year 2. In Year 4, for example, the program group earned $6,980 (including ze- 
roes for those who did not work), compared with $6,729 for the control group. As shown in Figure 
3.2, by Quarter 16, over 50 percent of both groups worked, and the program had no effect on this 
outcome. 


What might explain the difference between the two data sources? Each source has 
strengths and weaknesses. The survey data, for example, may suffer from response bias, if those 
who respond to a survey are different in some way from those who do not."° As noted in Chapter 
1, the NDNH records data contain quarterly wage data reported to the UI system, and these rec- 
ords miss some types of employment that are either exempt from reporting to the UI system (self- 
employment, domestic work) or underreported. Other research suggests that the UI data may miss 
relatively more employment for low-income populations than for higher-income groups. "! In ad- 
dition, smaller employers and employers with high turnover, who tend to employ relatively high 
numbers of less-skilled workers, tend to underreport earnings to the UI system more than other 
types of employers. '? 


The research team conducted several additional analyses to determine the cause of the 
difference between the data sources. First, effects using NDNH records data were estimated for 
the survey sample, to assess the possible effect of response bias. As shown in the Table B.6, 
impacts on records data earnings in Year 4 were larger for survey respondents than for the full 
sample ($458 for the survey sample versus $251 for the full sample), although neither difference 


°U.S. Department of Labor, Bureau of Labor Statistics (2017c). 

10s noted in Chapter 1, the survey achieved a response rate of 78 percent. 
11 \braham, Haltiwanger, Sandusky, and Spletzer (2013). 

Burgess, Blakemore, and Low (1998). 
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Figure 3.2 


Employment Rates by Quarter 
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SOURCE: Calculations based on data from the National Directory of New Hires. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 
10 percent. 
Sample size = 3,878. 


was Statistically significant. Differences in employment rates, in contrast, were very similar for 
both samples. These findings suggest that the more positive effects observed for the survey sam- 
ple (on both survey and records outcomes), compared with the full sample, explain at least in part 
the difference between the data sources. 


Second, the research team examined job characteristics for two groups of respondents: 
those who reported working on the survey and were also found to be working based on the UI 
data, and those who reported working on the survey but were not found in the UI data. There were 
several notable differences. Those whose survey-reported work was not found in the UI data were 
more likely to be self-employed (27 percent versus 17 percent) and more likely to be working in 
construction (13 percent versus 5 percent). In addition, among workers who were not self-em- 
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ployed, those whose reported employment did not match UI data received fewer employer-pro- 
vided benefits. These findings suggest that the difference between the two data sources can be 
explained in part by the fact that the UI records do not capture self-employment and may miss 
cash or intermittent jobs that are not reported by employers. 


YouthBuild’s effects are on par with effects found for Job Corps and ChalleNGe. Job 
Corps, for example, led to a similar-sized increase in survey-reported employment (of about 2.5 
percentage points) but a smaller increase in weekly earnings (of about 9 percent). However, one 
difference between the two studies is that the Job Corps evaluation also found positive, albeit 
much smaller, impacts on earnings based on records data reported to the Internal Revenue Service 
(primarily W2 forms). The effect on earnings in Year 4 using these data was a statistically signif- 
icant $218. For YouthBuild, there is no evidence from the UI data to suggest an increase in earn- 
ings, given that the difference of $251 in Year 4 is not statistically significant. However, the sam- 
ple size for the YouthBuild study (approximately 4,000 participants) was considerably smaller 
than that for the Job Corps study (about 15,000 participants). Separate calculations suggest that 
the difference would have been statistically significant if the sample size for YouthBuild were 
equal to that of Job Corps. 


ChalleNGe led to an increase in the employment rate of 4 percentage points during Year 
3, an increase in the employment rate of 7.1 percentage points at the time of the survey, and an 
increase in weekly earnings of 14 percent. YouthBuild’s effects on weekly earnings are slightly 
larger, with an increase of about 19 percent. 


Impacts on Youth Development 


Findings from the interim report showed that YouthBuild had increased young people’s civic 
engagement, primarily through an increase in volunteering. There were no effects on other aspects 
of youth development, as measured by various scales designed to capture social trust and self- 
esteem, for example. As noted in the earlier report, these aspects of youth development are diffi- 
cult to measure, and other research has found that they are also difficult to affect over the long 
term. Although it is unlikely that effects on these same measures will emerge anew at the 48- 
month point, early effects on educational attainment and college enrollment might lead to some 
effects. Therefore, this section presents impacts on civic engagement and youth development a 
full four years after study enrollment. 


Table 3.3 presents the results for selected outcomes using data from the survey. The re- 
sults match the earlier findings in showing a continued effect on volunteering — 58 percent of 
program group members reported volunteering since study entry, compared with 37 percent of 
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Table 3.3 
Impacts on Youth Development at 48 Months 


YouthBuild Control Difference 


Outcome Group Group (Impact) P-Value 
Civic engagement since random 
assignment (%) 94.3 90.6 3.7*** 0.000 
Volunteered 57.6 36.9 20.7*** 0.000 
Registered to vote® 87.8 88.1 -0.4 0.792 
Voted 53.6 49.5 4.1** 0.039 
Involved in politics or local community activi- 
ties 18.7 16.8 2.0 0.233 
Overall good health (%) 81.1 80.0 1.2 0.495 
Believes most people can be trusted (%) 20.7 19.8 0.9 0.605 
Self-esteem score® 3.3 3.3 0.0 0.462 
Sample size (total = 2,721) 1,784 937 


SOURCE: MDRC calculations based on responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


“Currently registered to vote at the time of the survey. 


Self-esteem is measured using the 10-item Rosenberg Self-Esteem scale. Response categories range 
from 1 = “strongly disagree” to 4 = “strongly agree,” where higher scores indicate higher levels of self-es- 
teem. The 10 items are averaged. 


control group members. Most of this increase in volunteering occurred during participation in 
YouthBuild. As noted in Chapter 1, programs receiving Corporation for National and Community 
Service funding, called YouthBuild AmeriCorps programs, strongly emphasize service to the 
community and participants receive education awards for completing the required number of ser- 
vice hours. 


The program had few other significant impacts related to youth development. The follow- 
up survey asked a number of questions designed to capture aspects of development such as self- 
esteem, self-confidence, depression and happiness, and orientation toward the future. The pro- 
gram had no consistent or lasting effects on these measures. Two measures are presented in Table 
3.3 and the remaining measures are shown in Appendix Table C.3. 
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Impacts on Involvement in the Criminal Justice System and Other 
Outcomes 


YouthBuild has the potential to affect a variety of youth outcomes, the most proximate being 
education, work, earnings, and youth development. By affecting these primary outcomes, the 
program may also have effects on other aspects of participants’ lives. This section examines ef- 
fects on these other outcomes. The interim report found few effects overall on these measures, 
with the exception of a small, positive effect on substance abuse. 


Table 3.4 presents the results for involvement with the criminal justice system and risky 
behavior. The program had no effect on arrest rates, conviction rates, or substance abuse. About 
30 percent of the study sample had been arrested since random assignment, which is not much 
higher than the rate reported at 30 months. The apparent leveling off of arrest rates may reflect 
the aging of the sample. In addition, 17 percent of the control group had been convicted since 
study entry, compared with 14 percent at the 30-month point, with the most common offenses 
being drug and property offenses. 


Rates of involvement with the criminal justice system for the YouthBuild sample are 
roughly similar to those for Job Corps, in which about 29 percent of control group participants 
had been arrested in the four-year follow-up period, and 22 percent had been convicted. A recent 
study estimated that, nationally, about 30 percent of young adults had been arrested by the age of 
23.8 Given the well-known relationship between educational level, race, and involvement in the 
criminal justice system, the arrest rate for black and Latino young adults without a high school 
diploma is probably much higher than that, suggesting that rates for the YouthBuild sample may 
be relatively low for this population. '* This finding is, perhaps, not surprising, given the high 
motivation level of the sample. 


Finally, Table 3.4 shows few differences between the research groups by the 48-month 
point. There was a very small increase in driving under the influence (of 1.7 percentage points) 
but the negative findings from the 30-month follow-up point — on binge drinking and other drug 
use — were no longer present. 


Appendix Table C.4 presents effects on other outcomes, such as living arrangement and 
family structure at the 48-month follow-up point. Overall, there are no notable effects. A com- 
parison of these data with data from the 30-month survey shows the study sample successfully 
moving into adulthood over time. For example, fewer young people in both the program and 


‘3Brame, Bushway, Paternoster, and Tumer (2014). This rate represents the percentage of young adults who 
were ever arrested by age 23, which is not exactly the same measure as reported in Table 3.4. 


Pettit and Western (2014). 
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Table 3.4 
Impacts on Delinquency and Risky Behavior at 48 Months 


YouthBuild Control Difference 


Outcome (%) Group _ Group (Impact) P-Value 

Arrested since random assignment 32.0 31.3 0.7 0.713 

Convicted or found delinquent since random 

assignment 19.8 17.4 2.5 0.131 
Drug offense 6.5 5.2 1.2 0.224 
Driving under the influence 1.6 1.5 0.1 0.788 
Failure to pay child support 0.3 0.3 0.0 0.895 
Property offense 6.0 5.6 0.3 0.735 
Violent offense® 5.1 3.8 1.3 0.152 
Other 3.0 2.5 0.5 0.525 

Incarcerated due to sentence since random assignment 11.7 10.8 0.9 0.510 


Substance abuse 


Has 5+ drinks once or more in typical week 28.9 29.3 -0.5 0.810 
Used marijuana since random assignment 50.1 51.3 -1.2 0.582 
Used another drug since random assignment 16.9 14.9 2.0 0.220 
Drove a car while drinking or doing drugs 
in the last 30 days 5.9 4.2 17% 0.085 
Sample size (total = 2,721) 1,784 937 


SOURCE: MDRC calculations from responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


“In the juvenile justice system, the term “adjudicated delinquent” is used rather than “convicted.” 


Property offenses include shoplifting, burglary, larceny, theft, auto theft, writing bad checks, fraud, for- 
gery, arson, vandalism, and possession of stolen goods. 


“Violent offenses include physical or sexual assault, rape, robbery, manslaughter, attempted murder, 
and murder. 


control groups reported living with their parents by the four-year point (40 percent at the 48- 
month point, compared with 50 percent at the 30-month point). A slightly higher proportion of 
young people in the sample were also married by this point, and more of them reported having 
children. 
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Conclusion 


In sum, the findings through 48 months show continued positive effects on educational 
attainment. YouthBuild led to an increase in high school equivalency credential attainment of 11 
percentage points and an increase in college enrollment of between 5 and 9 percentage points, 
depending on the source. Although the effect on college enrollment is encouraging, it occurred 
early in the follow-up period and did not persist into the fourth year. Effects on employment were 
positive, but were found primarily in survey-reported employment and earnings. Although the 
findings do not indicate a positive impact on UI-reported earnings, the trend shows an increasing 
difference over the four-year period. Finally, as reported in the interim report, the program had 
few effects on youth development, involvement with the criminal justice system, or other out- 
comes. The next chapter examines whether the effects of the program varied across types of par- 
ticipants and across types of programs. 
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Chapter 4 


Effects by Participant and Program Characteristics 


The previous chapter presented the overall effect of YouthBuild, across all young people and 
programs in the study. While this average effect provides a summary assessment, it may mask 
important variation in effects across different types of participants and across different types of 
programs. This chapter examines how YouthBuild’s effects vary across types of young people, 
defined by age, educational level, and gender. Although the young people who enrolled in the 
study are all disadvantaged and share many common challenges, they differ in many ways, and 
it is important to assess whether the program generally works better or the same for certain types 
of young people. The findings presented in this chapter help to put the full sample effects in con- 
text, and they have implications for whether programs might want to target resources or enroll- 
ment to certain groups of young people. Findings from the interim report showed some differ- 
ences among young people, with larger effects on employment for those who had left high school 
before completing tenth grade. This chapter updates these findings through 48 months. 


The chapter also presents an analysis of how program-level impacts vary with certain 
program characteristics. These characteristics include fidelity to the YouthBuild model as one 
measure of programmatic strength. However, even among programs with high fidelity, the pro- 
cess report documented variation across programs in how they implemented YouthBuild. Specif- 
ically, the chapter examines the following factors: length of Mental Toughness Orientation 
(MTO), whether the program requires a minimum reading and math score for entry, whether the 
program manages its own construction site, whether the program offers a high school diploma or 
equivalency credential track, program longevity, fidelity rating for the postsecondary educational 
component, and fidelity rating for the career development component. This analysis is nonexper- 
imental, since programs that have a particular feature may differ from other programs in a number 
of ways not accounted for in the analysis. Nonetheless, the findings point to suggestions for pro- 
gram improvement and further research. 


In sum, the findings show that YouthBuild’s effects varied somewhat across types of 
participants and programs. Effects on work and earnings tended to be larger for less-educated 
young people and for women, although these differences were not always statistically significant. 
Similarly, certain program characteristics were associated with larger effects — in particular, fea- 
tures related to how programs screen young people for YouthBuild and the strength of programs’ 
postsecondary educational services. 
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Effects for Subgroups of Young People 


The group of young people enrolled in the study was diverse. About 20 percent of study partici- 
pants were older than 21 years of age when they entered the study, for example, and a third were 
ages 16 to 18. Some young people nearly finished high school before dropping out, while others 
left school in the ninth or tenth grade. It is reasonable to expect that YouthBuild’s effects might 
vary for the different types of young people it serves. The selection of subgroups was informed 
by underlying theory or previous evidence about how certain characteristics might affect how 
young people interact with or benefit from the program. This section examines how impacts vary 
by the age, gender, and educational level of participating young people. 


The research team conducted the analysis by estimating impacts separately for each of 
the two paired subgroups and then assessing the differences in effects. In general, impacts are 
expected to vary to some extent between subgroups, simply as a result of natural variation around 
the average impact for the full sample. The analysis assesses whether that variation in impacts 
across subgroups is statistically significant, or greater than what would be expected to occur by 
chance alone. For that reason, the important question is not whether a given impact for, say, the 
younger subgroup is statistically different from zero, but whether that impact is statistically dif- 
ferent from the impact for the older subgroup (indicated by daggers in the rightmost column of 
the tables). If the difference between the impacts of these two subgroups is not statistically sig- 
nificant, the results suggest that the effects observed for the full sample generally hold across both 
comparison groups. 


Age 


During site visits, program staff reported that older participants tended to be more ready 
to benefit from YouthBuild, particularly relative to those age 18 or younger. In addition, the Job 
Corps evaluation found that the program’s effects on earnings persisted longer for those who were 
older when they entered the study.! On the other hand, older participants would have been out of 
school longer and may have found it more challenging to engage in an educational program. For 
the analysis, the research team divided the sample into study participants under age 20 when they 
entered the study and those age 20 or older when they entered. At 30 months, the only difference 
between the groups was that the older group had a larger effect on high school equivalency cre- 
dential receipt. 


Table 4.1 presents the results through 48 months. Overall, the effects were similar by age. 
Both groups, for example, had impacts of about 8 percentage points on postsecondary education 


‘Schochet, Burghardt, and McConnell (2008). 
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Table 4.1 
Impacts by Age at 48 Months 


Sample Members Under 20 Years Old Sample Members 20 Years Old and Older _ Difference 


Between 
YouthBuild Control Difference YouthBuild Control Difference Subgroup 
Outcome Grou Grou Impact P-Value Grou Grou Impact P-Value Impacts 


Education (%) 


Has a high school equivalency 
credential 33.2 24.3 8.9*** 0.001 35.7 23.0 12.7*** 0.000 
Ever enrolled in college since random 


assignment 
Based on survey data 26.5 21.2 5.3%* 0.046 27.1 22.5 4.6* 0.072 
Based on administrative data 21.4 13.3 8.1*** 0.0 21.1 12.3 8.7*** 0.000 


Youth development 


Voted since random assignment (%) 48.8 44.4 45 0.125 58.5 54.3 4.2 0.139 
Self-esteem score 3.3 3.3 0.0 0.819 3.3 3.3 0.0 0.642 
Employment 

Currently employed (%) 50.5 45.6 4.9 0.114 51.2 47.4 3.8 0.191 
Earning $10 per hour or more (%) 27.0 25.2 1.8 0.506 28.1 20.9 7.2*** 0.005 
Ever employed in Year 4° (%) 72.3 72.8 -0.5 0.833 70.8 73.6 -2.8 0.200 
Average earnings in Year 4° ($) 6,955 6,398 557 0.206 7,006 7,038 -32 0.939 
Ever arrested since random assignment (%) 33.7 31.3 2.4 0.398 30.1 31.4 -1.4 0.605 
Sample size (total = 2,721) 920 446 864 491 


(continued) 


Table 4.1 (continued) 


SOURCES: MDRC calculations using data from the National Directory of New Hires (NDNH), the National 
Student Clearinghouse, and responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The H-statistic is used to assess whether the difference between subgroup impacts is statistically signifi- 
cant. Statistically significant differences between subgroups are indicated as follows: ttt = 1 percent, tt =5 
percent, t = 10 percent. 

Unless otherwise indicated, all calculations are based on data from the 48-month survey. 


“Self-esteem is measured using the Rosenberg Self-Esteem scale. Response categories range from 1 = 
"strongly disagree" to 4 = "strongly agree,” where higher scores indicate higher levels of self-esteem. Re- 
sponses to the 10 items are averaged. 


These outcomes are based on NDNH data, not survey responses. The sample sizes for the younger pro- 
gram and control subgroups are 1,414 and 583, respectively. The sample sizes for the older program and 
control subgroups are 1,248 and 633, respectively. 


enrollment from the National Student Clearinghouse (NSC) records data. Effects on high school 
equivalency credential receipt were also similar (8.9 percentage points versus 12.7 percentage 
points), as were effects on employment reported on the survey. 


Highest Grade Completed 


As discussed in the process report, staff members reported that many young people en- 
tered the study with middle school reading and math levels, and a number of programs used as- 
sessments of basic skills to screen out very low-skilled candidates. On the one hand, young people 
with more education may be in a better position to take advantage of YouthBuild’s services — 
passing high school equivalency exams and benefiting from vocational training — suggesting 
larger impacts. On the other hand, these young people might do fairly well even in the absence of 
YouthBuild, so there might not be much difference between the program and control groups’ 
outcomes. The sample was divided into those young people who had completed the tenth grade 
or lower before leaving school, and those who completed the eleventh grade or higher. Note that 
the latter group included the 9 percent of sample members who had a high school diploma or 
equivalency credential when they entered the study. The interim report found differences in im- 
pacts on employment at 30 months, with the less-educated group having larger effects on em- 
ployment rates, measured using the survey and the National Directory of New Hires (NDNH) 
records data. 


Table 4.2 updates the results. By 48 months, there were still larger effects for the less- 
educated group, but only according to survey data. The program led to a 7.3 percentage point 
increase in survey-reported employment rates for the less-educated group, compared with a -0.2 
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Table 4.2 


Impacts by Highest Grade Completed at 48 Months 


Sample Members Who Had Completed Sample Members Who Had Completed __ Difference 
10th Grade or Lower 11th Grade or Higher Between 
YouthBuild Control Difference YouthBuild Control Difference Subgroup 
Outcome Group Group __(Impact) P-Value Group Group _(Impact) P-Value Impacts 
Education (%) 
Has a high school equivalency 
credential 35.8 24.9 10.9*** 0.000 33.2 19.7 13.5*** = 0.000 
Ever enrolled in college since random 
assignment 
Based on survey data 23.8 17.1 6.8*** 0.005 30.4 26.7 3.7 0.197 
Based on administrative data 17.7 10.6 7.16" 0.000 25.6 15.5 10.0*** 0.000 
Youth development 
Voted since random assignment (%) 48.8 46.1 2.7 0.328 59.9 53.3 6.5** 0.032 
Self-esteem score* 3.2 3.3 0.0 0.299 3.3 3.3 0.0 0.966 
Employment 
Currently employed (%) 48.8 41.5 7.3** 0.013 52.6 52.7 -0.2 0.958 ai 
Earning $10 per hour or more (%) 24.0 19.4 4.7* 0.057 30.8 28.1 2.7 0.360 
Ever employed in Year 4° (%) 69.6 71.8 -2.1 0.331 74.0 75.4 -1.5 0.517 
Average earnings in Year 4° ($) 6,364 6,239 125 0.753 7,752 7,386 366 0.443 
Ever arrested since random assignment 
% 34.4 32.5 2.0 0.483 29.2 30.3 -1.2 0.681 
Sample size (total = 2,680) 978 487 777 438 


(continued) 


Table 4.2 (continued) 


SOURCES: MDRC calculations using data from the National Directory of New Hires (NDNH), the National 
Student Clearinghouse, and responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The H-statistic is used to assess whether the difference between the subgroup impacts is statistically sig- 
nificant. Statistically significant differences between subgroups are indicated as follows: ttt = 1 percent, TT 
= 5 percent, t = 10 percent. 

Unless otherwise indicated, all calculations are based on data from the 48-month survey. 


“Self-esteem is measured using the Rosenberg Self-Esteem scale. Response categories range from 1 = 
“strongly disagree" to 4 = "strongly agree," where higher scores indicate higher levels of self-esteem. Re- 
sponses to the 10 items are averaged. 


*These outcomes are based on NDNH data, not survey responses. The sample sizes for program and 
control subgroup members who completed tenth grade or lower are 1,445 and 631, respectively. The sample 
sizes for program and control subgroup members who completed eleventh grade or higher are 1,177 and 
571, respectively. 


percentage point difference for the more-educated group. One hypothesis is that the employment 
effects might be lower for the more-educated group if more of them stayed in school longer, 
including some form of postsecondary education, instead of working. Impacts on NSC-reported 
college enrollment were larger for the more-educated subgroup. However, as the impacts on en- 
rollment fell over time for the full sample (see Figure 3.1 in Chapter 3), they also fell for both 
subgroups, with the result that impacts on college enrollment were zero for both groups by the 
end of the follow-up period (not shown). 


Another potential reason for the larger employment effects for the less-educated sub- 
group is simply the lower rates for the control group — only 41.5 percent of the less-educated 
control group reported that they were working at the time of the survey, compared with 52.7 
percent of their more-educated counterparts. In this case, there was more room for the program 
to have effects. However, the program did not increase employment for this group according to 
unemployment insurance records. 


Gender 


The challenges facing young men, especially African-American and Latino men, have 
been well documented and are evidenced by their high dropout rates and unemployment rates.” 
Young men stand to benefit the most from a program such as YouthBuild and might experience 
bigger effects than young women. On the other hand, the challenges they face may limit their 


U.S. Department of Education, National Center for Education Statistics (2014); U.S. Department of Labor, 
Bureau of Labor Statistics (2017b). 
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ability to take advantage of the program. Other evaluations have found that youth programs can 
have different effects on young men and women.* However, the interim report found no signifi- 
cant differences in effects for women versus men. 


Table 4.3 updates the results. Effects on education were similar across the two groups, 
particularly for college enrollment. Effects on employment and earnings reported on the survey 
are also not statistically different from young women versus young men. 


How Do Effects Vary Across Programs? 


Effects by Overall Fidelity 


The process study concluded that the evaluation provided a fair test of YouthBuild’s ef- 
fects, since most programs implemented the program reasonably well, or with high average fidel- 
ity. The overall fidelity rating is based on the research team’s assessments during site visits of 
over 60 of YouthBuild USA’s required design standards, or required elements of all YouthBuild 
programs. Research team members completed the fidelity rating tool after site visits, rating local 
programs on each standard as “meets standard,” “partially meets standard,” or “does not meet 
standard.” They could also choose “does not apply” or “unable to observe.” Point values were 
assigned to each standard, with programs receiving 100 points for “meets standard,” 50 points for 
“partially meets standard,” and 0 points for “does not meet standard.” To create the fidelity scores, 
the team summed the points across all standards for which there were observations, and then 
divided that number by the number of observations. The maximum possible rating was 100, and 
the minimum was 0. 


The average fidelity score was 79 out of 100, leading to the conclusion that most pro- 
grams implemented the program well. Nonetheless, there was some variation around this average, 
and a natural question that arises is whether the effects were larger in programs that had relatively 
high fidelity. This section presents effects in higher-fidelity versus lower-fidelity programs, where 
high-fidelity programs are defined as those achieving overall scores of 80 or higher. About half 
of the programs were above this cutoff, and these programs contained about 60 percent of survey 
respondents. At the 30-month point, there were few differences in effects by fidelity level, with 
the exception of service receipt. Specifically, impacts on receipt of educational, job-related, and 
personal development services were much larger in the high-fidelity programs. 


3See, for example, Miller et al. (2005). 
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Table 4.3 


Impacts by Gender at 48 Months 


Women Men Difference 
Between 
YouthBuild Control Difference YouthBuild Control Difference Subgroup 
Outcome Group Group __(Impact) P-Value Group Group _(Impact) P-Value Impacts 
Education (%) 
Has a high school equivalency 
credential 30.9 23.3 7.6** 0.012 36.7 23.9 12.8*** 0.000 
Ever enrolled in college since random 
assignment 
Based on survey data 28.8 24.6 4.2 0.190 25.5 20.2 5.3** 0.020 
Based on administrative data 23.0 14.7 8.3*** 0.000 20.1 11.9 8.3*** 0.000 
Youth development 
Voted since random assignment (%) 60.4 57.4 3.0 0.343 49.2 45.0 4.2 0.103 
Self-esteem score 3.3 3.3 0.0 0.790 3.3 3.3 0.0 0.460 
Employment 
Currently employed (%) 48.4 42.0 6.4* 0.069 52.4 49.3 3.1 0.248 
Earning $10 per hour or more (%) 23.5 16.1 7.4** 0.011 30.0 27.3 2.8 0.262 
Ever employed in Year 4° (%) 72.7 74.0 -1.3 0.618 71.2 72.7 -1.5 0.440 
Average earnings in Year 4° ($) 6,029 6,004 25 0.955 7,544 7,091 453 0.266 
Ever arrested since random assignment 
% 20.1 18.2 1.8 0.519 39.3 40.5 -1.2 0.665 
Sample size (total = 2,717) 692 352 1,090 583 


(continued) 


Table 4.3 (continued) 


SOURCES: MDRC calculations using data from the National Directory of New Hires (NDNH), the National 
Student Clearinghouse, and responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The H-statistic is used to assess whether the difference between the subgroup impacts is statistically 
significant. Statistically significant differences between subgroups are indicated as follows: TTT = 1 per- 
cent, tt = 5 percent, t = 10 percent. 

Unless otherwise indicated, all calculations are based on data from the 48-month survey. 


“Self-esteem is measured using the Rosenberg Self-Esteem scale. Response categories range from 1 
= "strongly disagree” to 4 = "strongly agree," where higher scores indicate higher levels of self-esteem. 
Responses to the 10 items are averaged. 


These outcomes are based on NDNH data, not survey responses. The sample sizes for the female 
program and control subgroups are 955 and 424, respectively. The sample sizes for the male program and 
control subgroups are 1,703 and 789, respectively. 


Table 4.4 presents the results through 48 months. As with the earlier subgroup findings, 
effects on education and college enrollment were fairly similar across both types of programs. 
There was a pattern of larger employment effects in the higher fidelity programs, but this differ- 
ence between programs was not statistically significant. In separate analyses (not shown), the 
research team examined differences in effects by dividing programs into three groups, rather than 
two, for low, medium, and high fidelity. No differences were found in effects across the three 


groups. 


Effects by Selected Program Characteristics 


Although the effects did not differ significantly by overall fidelity, they might differ 
across programs for other reasons. This section examines whether program impacts are associated 
with particular program characteristics. It is easy to imagine that program impacts might vary 
across different types of programs. For example, some programs had longer MTOs than others, 
which might affect the types of young people who eventually enroll in YouthBuild and the level 
of cohesiveness of the entering class. Another example is that some programs managed their own 
construction sites, while others contracted these services to an outside provider. Although both 
options fit within the YouthBuild model, earlier reports on YouthBuild suggested that the former 
arrangement provided programs with more control over the vocational training. 


As stated earlier, this analysis is nonexperimental. For example, the hypothetical finding 
that program impacts are larger in programs that run longer MTOs does not imply that longer 
MTOs cause larger impacts, since these programs might differ from other programs in ways that 
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Table 4.4 


Impacts by Program Fidelity at 48 Months 


Sample Members from Programs with Sample Members from Programs with Difference 
Low Program Fidelity High Program Fidelity Between 
YouthBuild Control Difference YouthBuild Control Difference Subgroup 
Outcome Group Group _(Impact) P-Value Group Group _(Impact) P-Value Impacts 
Education (%) 
Has a high school equivalency 
credential 37.5 26.0 11.5*** 0.000 32.5 21.8 10.6*** 0.000 
Ever enrolled in college since random 
assignment 
Based on survey data 29.6 24.8 4.8 0.103 25.3 19.3 6.0** 0.011 
Based on administrative data 24.3 16.1 8.2*** 0.000 19.2 10.2 9.0*** 0.000 
Youth development 
Voted since random assignment (%) 50.9 46.5 4.3 0.169 55.6 51.1 4.5* 0.083 
Self-esteem score* 3.3 3.3 0.0 0.567 3.2 3.3 0.0 0.579 
Employment 
Currently employed (%) 50.2 47.3 2.9 0.377 51.3 45.6 5.r* 0.040 
Earning $10 per hour or more (%) 30.3 25.8 4.5 0.128 25.7 21.0 4.6* 0.052 
Ever employed in Year 4° (%) 70.8 73.3 -2.5 0.287 72.2 73.1 -0.9 0.669 
Average earnings in Year 4° ($) 7,023 7,200 -177 0.707 6,940 6,378 562 0.154 
Ever arrested since random assignment 
% 29.8 28.9 0.9 0.765 33.6 32.7 0.8 0.741 
Sample size (total = 2,721) 686 404 1,098 533 


(continued) 


Table 4.4 (continued) 


SOURCES: MDRC calculations using data from the National Directory of New Hires (NDNH), the National 
Student Clearinghouse, and responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The H-statistic is used to assess whether the difference between the subgroup impacts is statistically sig- 
nificant. Statistically significant differences between subgroups are indicated as follows: ttt = 1 percent, tt 
= 5 percent, t = 10 percent. 

Unless otherwise indicated, all calculations are based on data from the 48-month survey. 


“Self-esteem is measured using the Rosenberg Self-Esteem scale. Response categories range from 1 = 
"strongly disagree" to 4 = "strongly agree,” where higher scores indicate higher levels of self-esteem. Re- 
sponses to the 10 items are averaged. 


These outcomes are based on NDNH data, not survey responses. The sample sizes for the low fidelity 
program and control subgroups are 1,076 and 540, respectively. The sample sizes for the high fidelity pro- 
gram and control subgroups are 1,586 and 676, respectively. 


are not accounted for in the analysis. Because programs were not randomly selected to implement 
various characteristics, such as a longer MTO versus a shorter MTO, the findings are suggestive 
only. 


Although a large number of factors might be associated with differences in impacts across 
programs, the analysis focuses on seven program characteristics. The reason for limiting the num- 
ber is based on the statistical power of the analysis, given the number of programs and young 
people in the study sample. The analysis focused on characteristics that: (1) were identified by 
the research team as potential drivers of program impact, (2) exhibited variability across programs 
(if nearly all programs had a particular characteristic, then it cannot explain variation in impacts), 
and (3) were reasonably measured with the data that were collected during the process study site 
visits. Based on these criteria, the research team selected the following seven characteristics (also 
shown in Table 4.5): 


Screening 


e Length of MTO, in days. Length of MTO is included in the model to capture how 
intensive the program was in terms of upfront screening. A higher level of screening 
might lead more young people to drop out before formally enrolling in YouthBuild. 
This might have two types of effects on program impacts. First, higher dropout rates 
from MTO can lead to fewer young people in the program group enrolling in Youth- 
Build, which might dilute program impacts. Second, the young people who eventu- 
ally do make it to YouthBuild after a longer MTO might be an especially motivated 
group. The effect on program impacts of serving more motivated young people is 
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Table 4.5 
Means and Percentile Values of Program Characteristics 


Value at percentile 


Program characteristic Mean 25th 50% goth 
Screening 
MTO length (in days) 10.5 6.5 10.0 19.0 
Minimum TABE score required for entry (%) 39.0 -- -- -- 


Program components 
Program operates own worksite (%) 51.0 -- -- a 


HS diploma or equivalency credential track (%) 39.0 -- -- - 


Implementation fidelity* 
Career development fidelity score 82.1 71.4 92.9 100.0 
Postsecondary fidelity score 65.1 40.0 70.0 100.0 


Program stability 
Years operating YouthBuild 10.4 6.0 9.0 18.0 


Program sample size = 72 


SOURCE: MDRC calculations using program data collected for the implementation study. 


NOTES: MTO = Mental Toughness Orientation; TABE = Test of Adult Basic Education; HS = high school. 
Means are presented for all variables. Percentiles are presented for continuous variables. 


“Fidelity scores could range from 0-100. 


unclear. As Table 4.5 shows, the average length of MTO across all programs 
was 10.5 days, although there is significant variation, with 10 percent of pro- 
grams running MTO for 19 days or more. The research team converted this 
measure to standard deviations from the overall average before including it 
in the analysis. 


e Minimum Test of Adult Basic Education (TABE) score required. This charac- 
teristic is also included to capture the intensity of screening. About 40 percent of 
programs reported that they required young people to score above a certain cutoff on 
a basic education assessment tool. (For most programs, the minimum score was the 
sixth-grade level in reading and math.) Other programs also used the scores, but more 
to tailor the academic services to participants. 
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Program Components 


e Construction training. (Program manages the work site.) Partners were often 
used for construction training, in part to help defray costs of running a work site. 
Work site partnerships had one possible drawback, however. When YouthBuild pro- 
grams did not control their work sites, they were less able to pace work appropriately, 
customize training schedules, and provide the variety of tasks needed for participants’ 
optimal learning. A 2009 evaluation of YouthBuild programs by Social Policy Re- 
search Associates, for example, found that programs that controlled their own work 
sites could customize training to participant needs better and were better able to en- 
sure that participants gained experience on more varied types of construction-related 
activities.* A little over half of the programs managed their own construction work 
sites. 


e Academic services. (Program offered high school diploma or equivalency cre- 
dential tracks.) The research team suggested that programs that offered both high 
school diploma and high school equivalency credential options may have stronger 
academic services because they could better tailor the programming to individual 
young people. Thirty-nine percent of programs offered both high school diploma and 
equivalency credential tracks (a few offered only a high school diploma option), with 
the remaining 61 percent of programs offering only the high school equivalency cre- 
dential option. 


e Career development fidelity score. The process study suggested that the job devel- 
opment component was less developed among programs than the educational and 
training components. In the 30-month survey, participating young people also rated 
this component (“help finding a job”) the least favorably among all YouthBuild ser- 
vices. The research team suggested that the career development fidelity score would 
be the best measure of the quality of this component. This score is based on ratings 
of several items that relate to the support the program provides young people during 
their active enrollment in the YouthBuild program to pursue a career path of interest 
to them. The average fidelity score on this component was fairly high, at 82 out of 
100, although 25 percent of programs had scores of 71 or lower. 


e Postsecondary education fidelity score. Given the growing importance of post- 
secondary education, many YouthBuild programs have begun focusing on the tran- 
sition to college and making connections with local postsecondary institutions. The 
score includes ways in which the programs offered support from within the organi- 
zation or strategically partnered with institutions or other organizations to support the 


4Abrazaldo et al. (2009). 
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participants’ needs around postsecondary preparation, admissions applications, en- 
rollment, and bridge programs. The average fidelity score for this component was 
lower than for overall fidelity, at 65 out of 100. 


Stability 


Number of years the program has been operating. The process report documented 
that a number of programs closed their doors after serving young people in the study, 
and that many reported not having long-term funding plans. Funding instability, in 
particular, was an issue for many programs and hindered their ability to keep staff 
and improve programming. Absent a good measure of funding stability, the research 
team decided that the length of years the program has operated would capture the 
strength of the program and its stability. The average program had operated for 10 
years. The research team converted this measure to standard deviations from the 
overall average before including it in the analysis. 


Outcomes 


As with the selection of program characteristics, the statistical power of the test requires the anal- 
ysis to focus on a more limited set of key outcomes. They are the following: 


High school equivalency credential attainment, measured on the 48-month survey 
Ever enrolled in college since random assignment, from the NSC records data 
Working at the time of the 48-month survey interview 


Employed in Year 4, from the NDNH records 


The first step is to test whether impacts on these four outcomes vary across programs. If 
all programs had roughly similar impacts on high school equivalency credential attainment, for 
example, there would be little variation with respect to program characteristics to explain. When 
examining impacts for each individual program, the first test is whether the variation that exists 
is more than what would be expected simply by chance, or whether it is statistically significant. 
A separate analysis (discussed in Appendix B) finds that there is statistically significant variation 
across program impacts for each of the four key outcomes. 


Analysis 


This analysis differs from the subgroup analyses presented earlier. Instead of dividing the sample 
into two subsamples, the research team estimated effects in a model in which the outcome (for 
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example, high school equivalency credential receipt) is the dependent variable and is regressed 
on treatment status interacted with each of these program characteristics.° The coefficient on the 
interaction indicated whether impacts vary with respect to that particular characteristic. Separate 
tests indicated that the fidelity scores were associated with program impacts in a nonlinear way. 
For this reason, the research team entered the scores as dichotomous variables indicating a me- 
dium or a high value for the fidelity score, so that their interpretation is relative to a “low” fidelity 
score. 


The model also included the following community characteristics: whether the area is 
urban versus rural; the average local unemployment rate during the final three years of the follow- 
up period; and building permits relative to trend, to capture the strength of the housing market, 
also measured during the final three years of the follow-up period. The model took account of 
participant characteristics as well, including age, race, gender, and highest grade completed at 
study entry. All of these characteristics are also interacted with treatment status, to control for 
differences in local environment and the type of young people served. 


Table 4.6 presents the results of this analysis. The values listed are coefficients from the 
regression model and associated p-values, indicating whether that particular program character- 
istic had a statistically significant association with program impacts. The primary focus is on the 
sign (negative or positive) of the coefficient, rather than its size, given that the program charac- 
teristics are measured in different units. 


For example, requiring a minimum TABE score for program entry (the second row under 
“screening”) appears to have been associated with smaller impacts on survey-reported employ- 
ment. In other words, programs that required minimum scores tend to have smaller effects on 
employment than programs that do not require minimum scores. Requiring minimum scores may 
lead to a more work-ready group, meaning that the control group would be more likely to find 
work on their own, in the absence of the program. Note that this finding is consistent with the 
subgroup finding that the employment effects were larger for young people with lower education 
levels at study entry. The other screening characteristic, MTO length, does not appear to have 
been related to program impacts on any of the outcomes examined. 


As for the two program component characteristics, offering more than just a high school 
equivalency credential track was not associated with program effects. Programs that managed 
their own work sites, in contrast, appear to have had smaller effects on high school equivalency 


°In separate models, the research team entered program characteristics separately into the models, in addition 
to entering them as interactions with treatment status. The results from these models with “main effects” added 
were similar to those reported here. 


57 


8S 


Table 4.6 


Associations between Program and Community Characteristics and the 
Impact of YouthBuild on Participant Outcomes after 48 Months 


High School Equivalency —_ Enrolled in Postsecondary Employed in 
Credential Attainment Education (Records) Currently Employed Year 4 (Records) 
Estimated Estimated Estimated Estimated 

Program Characteristic Coefficient P-Value Coefficient P-Value Coefficient P-Value Coefficient P-Value 
Screening 

MTO length (in days)@ 0.77 0.734 -1.02 0.499 -0.84 0.701 -0.51 0.762 

Minimum TABE score required for 

entry -4.10 0.239 0.30 0.948 -8.11* 0.078 1.08 0.850 
Program components 

Program operates own worksite -6.88* 0.060 -2.52 0.430 6.04 0.183 1.46 0.733 

HS diploma or equivalency 

credential track -3.60 0.428 1.72 0.692 -5.25 0.296 -1.55 0.712 

Implementation fidelity 

Career development score medium -3.35 0.500 5.35 0.200 0.88 0.889 2.87 0.589 

Career development score high -6.37 0.134 -1.40 0.734 3.40 0.497 -0.22 0.969 

Postsecondary score medium -0.64 0.860 0.72 0.856 -6.98 0.285 -3.96 0.402 

Postsecondary score high 5.38 0.300 9.58** 0.028 -15.03** 0.023 -4.44 0.372 
Program stability 

Years operating YouthBuild@ -0.74 0.647 1.05 0.512 -2.61 0.124 -0.94 0.498 
Community characteristics 

Local unemployment rate? -1.58 0.193 1.16 0.271 -1.86 0.274 -2.55** 0.031 

Building permits relative to trend -0.03*** 0.008 -0.01 0.545 0.00 0.621 0.00 0.952 

Urban area -7.09* 0.061 -0.39 0.930 -2.19 0.741 -5.30 0.296 


Sample size (participants = 3,929, programs = 72) 


(continued) 


Table 4.6 (continued) 


SOURCES: MDRC calculations using data from the National Directory of New Hires (NDNH), the National 
Student Clearinghouse, responses to the 48-month survey, program data collected for the implementation 
study, the U.S. Census, and the U.S. Bureau of Labor Statistics. 


NOTES: MTO = Mental Toughness Orientation; TABE = Test of Adult Basic Education; HS = high school. 

Results in this table are regression-adjusted, controlling for individual baseline characteristics and site 
fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The estimated coefficients represent how the impacts on the listed outcomes vary with each program and 
community characteristic. The estimates were obtained by fitting an impact model that includes an indicator 
of treatment status, as well as a set of interaction terms between the treatment indicator and each of the pro- 
gram and community characteristics. The findings reported in the table are the coefficients of the interaction 
between treatment status and program and community characteristics. The model also controls for site fixed 
effects and individual baseline characteristics. 


“The length of time was converted to a standardized score (each site was assigned a value equal to the 
number of standard deviations below or above the average length of time across all sites) in order to make it 
easier to compare coefficients across variables. 


Local unemployment rate was calculated as an average of the last three years of follow-up, based on 
random assignment dates. 


“Building permits relative to trend was calculated by subtracting the average number of building permits 
issued in the 156-168 months before the beginning of the post-program period (depending on data availabil- 
ity) from the average number of building permits issued during the post-program follow-up period (the last 36 
months of follow-up). 


credential receipt and larger effects on work than other programs, although the latter association 
was not statistically significant. Perhaps these programs put less emphasis on academic instruc- 
tion, or maybe participating young people dropped out before earning a high school equivalency 
credential because they had better employment opportunities. Similarly, as shown in the next set 
of rows under “implementation fidelity,” programs with high career development ratings seem to 
have had smaller effects on high school equivalency credential receipt (a coefficient of -6.4 per- 
centage points, with a p-value of 0.134), potentially for similar reasons. 


A higher postsecondary fidelity rating was associated with larger effects on postsecond- 
ary enrollment. These programs with higher ratings also had smaller effects on survey-reported 
employment than other programs (the effect on NDNH-reported employment was negative and 
not statistically significant), suggesting that young people may have been substituting school for 
work or that programs with a strong postsecondary emphasis had fewer services to help young 
people find jobs. The process study found that programs that did not have strong postsecondary 
services often reported that participants were not interested in attending college, and they there- 
fore focused their efforts on placing young people into jobs. 


The local economy was somewhat associated with impacts on education and work. Pro- 
grams in areas with stronger local housing markets had smaller impacts on high school equiva- 
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lency credential receipt. Similarly, a higher unemployment rate was associated with smaller ef- 
fects on work, although only the association with NDNH-reported employment was statistically 
significant. This finding highlights the importance of the economy in interpreting YouthBuild’s 
effects. Although the Great Recession officially ended in 2009, unemployment rates fell very 
slowly during the recovery and remained relatively high for young adults, only falling below 10 
percent by the start of 2017. 


Programs in urban areas also tended to have smaller effects on high school equivalency 
credential receipt. Separate analyses indicate that program impacts on the receipt of educational 
services were generally similar in rural and urban areas, so it is not clear what was driving this 
association. 


In sum, these findings, although nonexperimental and not always statistically significant, 
show patterns in three areas. First, in terms of screening, requiring minimum academic scores for 
program entry was associated with smaller effects on employment. Taken together with the find- 
ings from the subgroup analysis showing more positive employment impacts for less-educated 
young people, the results suggest that programs might consider targeting resources towards those 
with lower educational and literacy levels. Or, at a minimum, programs may not want to screen 
these young people out, since they may stand to benefit the most from a program such as Youth- 
Build. 


Second, programs that had stronger postsecondary educational services tended to have 
larger effects on college enrollment and smaller effects on work. Given that today’s labor market 
increasingly requires a postsecondary credential of some kind, the focus on high-quality postsec- 
ondary services seems appropriate. However, the next step is to help young people who enter 
college stay enrolled and complete a degree. 


Finally, programs that had stronger employment services (in terms of managing their own 
work site or having better career development services) tended to have smaller effects on educa- 
tion (high school equivalency credential receipt) and larger effects on employment. 
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Chapter 5 


The Status of the YouthBuild Investment 


The previous two chapters assessed the impacts of YouthBuild programs on sample members’ 
education, employment, earnings, and other outcomes. This chapter uses a benefit-cost analysis 
to weigh the programs’ effects against their costs. Despite having four years of follow-up on the 
program and control groups, it is too early to draw conclusions from the analysis. After a direct 
outlay of more than $24,000 per YouthBuild participant,! the return on investment started with 
the completion of homes built in local communities and the entry of some participants into the 
labor market. However, this early return was partly offset by a further, indirect investment, as 
participants entered publicly supported colleges in large numbers. After four years, the investment 
in YouthBuild is nearly complete, but only a fraction of the return has been recorded. More time 
is needed to see if short-term effects translate into lasting economic improvement. 


While inconclusive, the benefit-cost assessment shows that YouthBuild is valuable to 
participants, but its net value to taxpayers and society depends on the size of earnings impacts 
beyond the four years covered by the evaluation. Under scenarios where YouthBuild’s measured 
impact on earnings drops sharply or holds firm in future years — either plausible given the avail- 
able research evidence — the value of the investment to taxpayers and society ranges from neg- 
ative to positive. An underlying issue is whether the additional postsecondary education received 
by the YouthBuild program group has the sustained effect on earnings it typically has for the 
general population. Without further follow-up of the sample, this issue cannot be settled. 


This chapter starts by examining the investment made in YouthBuild — both the direct 
program costs and the indirect costs arising from the program’s effect of increasing enrollment in 
postsecondary education. The discussion then turns to the return on this investment, which takes 
many years for a program such as YouthBuild. Only a small part of this return has been registered 
after four years, so it is important to assess how large the future return might be. In this context, 
the benefit-cost analytical framework and alternative assumptions about the future course of sam- 
ple members are used to gauge YouthBuild’s potential economic value. 


‘All dollars are presented here in 2017 dollars. 
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The Investment in YouthBuild 


The direct investment in YouthBuild by the federal government and by state and local agencies 
is the first topic of this section. The investment is captured by program costs incurred for the 
program group in 54 programs, a subsample of the 75 programs in the evaluation.* These costs 
and the sources used to pay them differ substantially across programs, as explained below and in 
Appendix D. The second topic is the indirect investment by community colleges and other post- 
secondary institutions, where YouthBuild transmitted many participants after they completed 
their secondary education and initial skills training. This investment was measured using data 
from surveys and school records, and with data on postsecondary educational costs from states 
where the YouthBuild programs operate. 


Direct YouthBuild Investment 


The cost of YouthBuild across the 54 programs is $24,521 per participant.* It is a large 
investment in a distinctive program. YouthBuild operates on a smaller scale than comparable 
programs and spends time and money building houses in communities as well as delivering sec- 
ondary education and skills training. Table 5.1 breaks this cost down by program function. The 
cost estimates in the table include all on-budget expenditures — that is, the amounts paid with 
funds from the U.S. Department of Labor (DOL) and the Corporation for National and Commu- 
nity Service (CNCS) and matching funds provided or secured by each program’s host organiza- 
tion and off-budget expenditures.* A little over half of on-budget expenditures are devoted to 
education- and training-related activities. More than a third is for other services (counseling, case 
management, service to community, and services other than education and training) and local 
program management. One of the other services, service to community, includes the construction 
of affordable housing and facilities in the vicinity of YouthBuild programs’ — a distinguishing, 
but expensive aspect of the program. 


As shown in Figure 5.1, across all programs, 46 percent of the cost of YouthBuild is paid 
by DOL and another 8 percent is covered by CNCS and other federal agencies. Six percent of the 


Cost data are not available for 21 programs, either because the data were not collected during the site visit 
or because the data collected were not complete. A separate analysis showed that the 54 programs with cost data 
were roughly similar to the programs without cost data, with the exception that the cost sites were somewhat less 
likely to be affiliates of YouthBuild USA. 

3This cost represents the cost per participant over the entire program period. 

4A program’s budget includes DOL and CNCS grants plus required matching funds. Any state and local 
funds (including private money) in excess of the matching funds in the budget are “off budget.” 

°Some of the cost of building affordable housing (for instance, supervision provided by construction trades 
instructors) is included in “training related services.” 
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Table 5.1 


Direct Cost by Service per Participant (2017 Dollars) 


Program Component Cost ($) Percentage of Total (%) 
Education-related services 5,517 22.5 
Job or training-related services 7,042 28.7 
Other services and local management 8,958 36.5 
Stipends 3,004 12.3 
Total Direct Cost 24,521 100 


SOURCES: MDRC calculations using cost data from 54 YouthBuild sites and the 2011 Youth- 
Build USA Annual Report. 


NOTES: Tests of statistical significance were not performed. 

Rounding may cause slight discrepancies in sums and differences. 

Program costs are based on one year of costs (typically fiscal year 2011-2012) and aim to 
capture the steady state of operation excluding external research costs. 

All costs are shown in constant 2017 dollars. 

Cost estimates were adjusted for inflation using the gross national product implicit price 
deflator. 


cost is paid by national private sources, including foundations, companies, and individuals. The 
remaining 38 percent is covered by local sources, which include a variety of state and local gov- 
ernment agencies (school districts, workforce boards, city offices, and so on) and a diverse group 
of private contributors. As mentioned in Chapter 1, programs receiving DOL grants are required 
to match 25 percent of their grant with nonfederal funds. The requirement is met with funds from 
partner organizations and, particularly for host organizations with multiple youth programs, cost 
sharing with other budgets. Matching accounts for considerably more than 25 percent of most 
YouthBuild budgets. 


The data underlying these cost estimates are from four sources. One is local programs, 
which supplied on-budget expenditure data for the 2011 and 2012 program years, when program 
group members were enrolled in YouthBuild. Another is management information system (MIS) 
data, available from programs receiving DOL funding, and used to verify or determine program 
enrollment in some cases. The third is interviews with local program staff members, which pro- 
vided off-budget contribution information as well as participation data for sites without MIS data. 
The last source is YouthBuild USA, which supplied data on stipends and education awards and 
administrative expenses. 
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Figure 5.1 


YouthBuild Costs by Funding Source 


Federal 57% 


Other federal 


State and local 


National private 
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Percentage of total program costs 


SOURCES: MDRC calculations using cost data from 54 YouthBuild sites and the 2011 YouthBuild 
USA annual report. 


NOTES: DOL = Department of Labor; CNCS = Corporation for National and Community Service. 
Cost estimates were adjusted for inflation using the gross national product implicit price deflator. 


YouthBuild’s Investment Compared with Similar Programs 


YouthBuild’s direct investment, $24,521 per participant, is larger than those of four somewhat 
comparable programs providing education, training, and supportive services to similar popula- 
tions. Job Corps makes the next largest investment, just under $24,500 per participant in 2017 
dollars.* A large residential program administered by DOL, Jobs Corps spends less than half as 
much on education and vocational training, less on stipends, and very little on community service 


°The costs for Job Corps come from McConnell and Glazerman (2001) and the costs of other programs are 
from Cave, Bos, Doolittle, and Toussaint (1993), Jastrzab, Masker, Blomquist, and Orr (1996), and Perez-Arce, 
Constant, Loughran, and Karoly (2012). Costs are adjusted to reflect 2017 dollars using the gross national prod- 


uct price deflator. 
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activities. However, Job Corps devotes more to other local program functions — primarily be- 
cause of the costs of providing room, board, health care and residential services — and also 
spends more on central administration of the program. 


Youth Corps, a CNCS program with nonresidential and residential sites, involves a direct 
investment of about $16,000 per participant. The Youth Corps has the same commitment to ser- 
vice to the community as YouthBuild, although the service usually involves teaching and human 
services instead of house building.’ However, about half of its participants enter programs with 
high school diplomas and some have already had extensive postsecondary education. Thus, the 
investment of Youth Corps programs in secondary education is smaller, and the types and extent 
of supportive services are usually different from those YouthBuild provides. 


The ChalleNGe program, operated in residential centers by the National Guard, has made 
a direct investment of about $14,000 per participant. ChalleNGe has a less-intense service to 
community focus than YouthBuild, but it serves a similar youth population, delivers comparable 
secondary education and support, and has virtually the same average length of participant enroll- 
ment. 


Jobstart, a demonstration program tested by MDRC in the 1980s, was an intervention 
designed to deliver nonresidential Job Corps-type services. Indeed, 3 of the 13 Jobstart sites were 
nonresidential Job Corps centers. Jobstart, which cost about $9,000 per participant in 2017 dol- 
lars, provided education, training, and support comparable to residential Job Corps Centers, but 
without room, board, or residential services. 


The chapter later describes the results of a cost-benefit analysis that the research team 
conducted for YouthBuild as well as findings from these other four programs. In addition to hav- 
ing similar objectives, the programs all have been rigorously evaluated in randomized controlled 
trials. The evaluations used relatively consistent methods for estimating the costs of serving pro- 
gram and control group members and for estimating impacts on the outcomes discussed in Chap- 
ter 3. The program and control groups were observed for four years in the Job Corps and Jobstart 
evaluations — the same as for YouthBuild — and for shorter but still comparable three-year 
periods in the evaluations of ChalleNGe and Youth Corps. 


Why is YouthBuild more expensive than the other programs? First, it provides secondary 
education to nearly all its participants. In addition, although the majority of programs offer high 
school equivalency preparation only, a significant share of programs deliver their educational 
services through charter schools, career academies, and locally developed high school arrange- 
ments. (See Table 5.2.) One-fourth of enrollees in Job Corps and Jobstart, and half of Youth Corps 


7Jastrzab, Masker, Blomquist, and Orr (1996). 
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Characteristics of Selected YouthBuild Programs 


Table 5.2 


Host Education Federal Number of Cost per 
Site Organization Location Delivery Funding (%) Participants _ Participant? ($) 
Site 1 Nonprofit Large metro, central Charter School, College Credit 23.7 214 22,011 
Site 2 Nonprofit Large metro, central Charter School 100.0 33 21,231 
Site 3 Nonprofit Large metro, central GED, College Credit 32.2 74 28,929 
Site 4 Nonprofit Medium metro GED 90.0 25 36,672 
Site 5 School district Medium metro High School, GED, Career Academy 96.8 40 20,609 
Site 6 Nonprofit Small metro High School/GED 73.7 35 23,461 
Site 7 Community college Rural GED, College Credit 74.4 27 22,837 
Site 8 Community college Medium metro GED, College Credit 100.0 20 22,820 
Site 9 Nonprofit Large metro, fringe Charter School 82.9 62 13,431 
Site 10 Public agency Large metro, central Career Academy, College Credit 68.3 40 28,159 
Site 11 Nonprofit Rural GED 67.3 25 28,521 
Site 12 Nonprofit Large metro, fringe High School, College Credit 79.2 23 36,069 
Entire Study 83% nonprofit, 35% large central, 15% charter schools, 56.6 41 24,521 


17% other hosts 


17% fringe, 19% medium, 


8% small, 21% rural 


32% college credit 


SOURCE: MDRC calculations using cost and implementation data from 54 YouthBuild sites. 


NOTE: “Cost estimates were adjusted for inflation using the gross national product implicit price deflator. 


members, already possess a high school diploma or equivalency credential; and few of those who 
need it obtain a diploma from a charter school or special high school. YouthBuild also provides 
postsecondary education courses, leading to credentials and sometimes to college credit, and 
steers its graduates into college programs using college guidance and AmeriCorps education 
awards. Job Corps also provides postsecondary vocational credentials and Youth Corps offers the 
same education awards as YouthBuild. 


Second, YouthBuild has other characteristics that make it more expensive. One is the 
small scale of most local programs. On average, YouthBuild programs serve 41 participants per 
year, which is much smaller than the number in Job Corps and smaller than the numbers in Chal- 
leNGe, Jobstart, and Youth Corps.® There are scale economies in operating these programs, just 
as in running most other public and private enterprises. Consistent with these scale economies, 
most of the larger YouthBuild programs have a cost below the average of $24,521. 


Another reason YouthBuild is more expensive is that many of its programs are in major 
cities. Eighty percent of its programs are in metropolitan areas and more than a third are in large 
city centers. Only one-sixth of Job Corps centers are in large metropolitan areas, while none of 
the sites in the ChalleNGe evaluation were in city centers. (All but one were in rural areas.) The 
cost of YouthBuild programs in city centers is more than $2,000 above the average. Other reasons 
for YouthBuild’s high cost include the expenses of small nonprofits sponsoring many of its pro- 
grams and the building of affordable community housing, which includes the additional costs of 
construction supervisors, equipment, supplies, and materials. 


Net Program Costs per Sample Member 


The benefit-cost analysis starts with the value of YouthBuild’s net total investment per sample 
member — that is, the costs for program group members minus the estimated costs of comparable 
services obtained by the control group. The research team used two computations to calculate the 
net costs per sample member (presented in Table 5.3) from the expenditures per participant (pre- 
sented in Table 5.1). One is multiplying the costs by the participation rates for each research 
group, which yields gross — or total — costs per group member (shown in the bottom panel of 
Table 5.3). The second is subtracting the costs per control group member to get the differences in 
cost (shown in the table’s third column). Appendix D provides more details. 


®The evaluation of Job Corps included 38 centers serving (on average) 225 participants or less, 49 with 226 
to 495 participants, and 16 with 496 participants or more (Burghardt and Schochet, 2001). ChalleNGe sites 
served about 200 participants each year (Millenky, Bloom, Muller-Ravett, and Broadus, 2011) and the number 
of participants in the 13 Jobstart sites covered a wide range (Cave, Bos, Doolittle, and Toussaint, 1993). The 
evaluation of Youth Corps excluded local programs with fewer than 50 participants (Price et al., 2011). 
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Table 5.3 
Net Cost per Sample Member (2017 Dollars) 


Program Control Difference 
Feature Group Group (Net) 


Unit cost ($) 


Education-related services 5,517 2,220 3,297 
Job- or training-related services 7,042 4,363 2,679 
Other services and local management 8,958 3,676 5,282 
Stipends 3,004 0 3,004 


Participation rate (%) 


Education-related services 75.0 57.4 17.5 
Job- or training-related services 70.8 39.4 31.4 
Other services and local management 59.3 31.3 28.0 
Stipends 84.2 0.0 84.2 


Total cost ($) 


Education-related services 4,135 1,275 2,861 
Job- or training-related services 4,987 1,718 3,268 
Other services and local management 5,315 1,152 4,162 
Stipends 2,529 0 2,529 
Total cost 16,965 4,145 12,820 


SOURCES: MDRC calculations using cost data from 54 YouthBuild sites, the Legislative Ana- 
lyst’s Office 2012, the Massachusetts Budget and Policy Center 2017, the Boston Foundation 
2011, and responses to the 12-month survey. 


NOTES: Tests of statistical significance were not performed. 

Rounding may cause slight discrepancies in sums and differences. 

Analysis assumes that cost of education, vocational, and personal development services are 
equal for program and control group members. Assumes control group members do not receive 
stipends. 

Cost estimates were adjusted for inflation using the gross national product implicit price 
deflator. 


The total net cost of the YouthBuild programs, $12,820 per sample member, is lower 
than the gross cost, a reminder that the control group obtained its own services. It is particularly 
noteworthy that 57 percent of control group members obtained educational services. 
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Indirect YouthBuild Investment 


About 27 percent of the program group reported having enrolled in postsecondary education 
courses during the follow-up period. Some of this enrollment is the result of YouthBuild’s direct 
investment, but most represents an indirect investment made after participants left the program. 
The cost of this effect on post-program postsecondary education depends on the state where each 
student enrolled. In California, home to one-tenth of the program group, the lion’s share of en- 
rollment was in community colleges, where the cost of full-time study was about $7,000 per year 
in 2017 dollars,° higher than the average cost of full-time study across all degree-granting two- 
year institutions nationwide. '° Program group members paid some of this cost in tuition and fees, 
sometimes using their education awards from YouthBuild AmeriCorps funding, and the remain- 
der was paid with state and local tax revenue. 


Across the entire sample, the estimated total cost of the indirect investment is $928 per 
program group member — about one-fourth of the cost of YouthBuild’s direct investment in 
secondary and postsecondary education. An estimated 6 percent of this cost was paid with edu- 
cation awards received by program group members who participated in YouthBuild programs 
affiliated with AmeriCorps. !! The net cost is $317 per program group member, reflecting impacts 
of 2.8 percentage points on full-time study (at a cost of $5,907 per full-time student) and 8.4 
percentage points on part-time study ($1,949 per part-time student). For a further explanation of 
the gross and net cost calculations, see Appendix D. 


Many sample members were not in the labor market, or worked part-time rather than full- 
time, while they studied. This imposes an opportunity cost on both participants (who forego 
wages) and the economy (which forgoes output), something that is reflected in the evaluation’s 
employment and earnings results over the four-year observational period. 


Return on Investment 


The return on investment (ROJ) in YouthBuild takes place over participants’ lifetimes. The first 
two stages of YouthBuild’s return — during the time program group members participate in 
YouthBuild programs and in the first three or so years after they leave — occurred during the 
four-year observational period and can be measured with confidence. The return in subsequent 


°California Community Colleges Chancellor’s Office (2017). 
10U.S. Department of Education, National Center for Education Statistics (2016b). 


11 third of participants in the treatment group received awards from AmeriCorps, and the average award 
was $1,871. Data on the use of these awards through the end of the observation were not available. However, it 
was assumed that about 10 percent of the awards were used. The remainder was available for use in future years. 
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years must be estimated using assumptions. The benefit-cost analysis addresses all three stages, 
using alternative sets of assumptions for the third. 


Nearly all the tangible YouthBuild ROI consists of increased economic output — the 
goods and services program group members help produce during and after their time in the pro- 
gram. This output is the focus of the following discussion of ROI for YouthBuild and the other 
four programs. When the discussion turns to the benefit-cost analyses of the programs, other com- 
ponents of the return are considered. 


Year 1 Return 


The construction of affordable housing and facilities in local communities, one of the 
hallmarks of YouthBuild, is the primary component of the return on investment in the first year 
following random assignment. This construction is valued in much the same way comparable 
community service has been in the benefit-cost analyses of Job Corps, Youth Corps, and Chal- 
leNGe. Data collected by YouthBuild USA indicate that program group members devoted an 
average of 326 hours to community service during their time in the program. This number varied 
greatly among the local YouthBuild programs: Young people assigned to 6 programs, including 
2 of the largest programs in the evaluation, devoted just under 600 hours per program group 
member, while participants in 17 other programs spent less than 100 hours each. 


The hours of service per program group member in YouthBuild is less than the average 
of 600 in Youth Corps, which includes a range of volunteer service programs included in the 
AmeriCorps Network. '? However, it is much greater than the 66 in ChalleNGe, the 31 in Job 
Corps, and small and unmeasured number of hours in Jobstart.'° 


The research team created an estimate of the value of the YouthBuild service by assuming 
community services would not have been performed in the absence of the program, valuing ser- 
vice hours at the lowest-paid residential construction workers, and adding the cost of construction 
materials to the value of the work. (See Appendix D for details.) The resulting estimate of the 
value of the work is $7,867. In the benefit-cost analysis presented below, the research team treated 
this dollar value as a benefit to taxpayers and to society. 


Most of the construction, with a value of $6,557, was completed in the first year following 
random assignment. The value of other output produced by the program group in the first year is 
indicated by their wages ($2,093) and fringe benefits ($563), the latter estimated using data from 


12Jastrzab, Masker, Blomquist, and Orr (1996). 
'3Perez-Arce, Constant, Loughran, and Karoly (2012); McConnell and Glazerman (2001). 
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the U.S. Bureau of Labor Statistics.'* The sum of these estimates, $9,213, is the total value of the 
goods and services produced by the program group in the first year after random assignment. 


To calculate the net first-year return on the YouthBuild investment, the wages and bene- 
fits of the control group during that period ($3,055) must be subtracted from the $9,213 in output 
produced by the program group. Thus, the net first-year return is output worth $6,159, or 46 per- 
cent of the investment in the program. 


Return in Years 2 Through 4 


The second ROI phase began after program completion, when program group members 
mixed postsecondary education with full- and part-time employment as they made the transition 
into adulthood. Eighty percent of program group members were enrolled in YouthBuild for less 
than 12 months, which means this second phase began sometime in their first year after random 
assignment. The remaining 20 percent stayed enrolled into their second year after random assign- 
ment, so part of the ROI in Years 2 through 4 comes from the value of YouthBuild construction 
from this segment. The estimated value of this work is $1,157. In addition, the earnings and fringe 
benefits of the full program group in Years 2 through 4 ($16,157 in earnings and $4,347 in bene- 
fits) exceed those of the control group by $204, lifting the total return to $1,361. However, the 
net cost of the indirect investment in postsecondary education must be subtracted from this return, 
leaving a net return of $1,044 during the three years, or 8 percent of the net investment in Youth- 
Build. 


Thus, YouthBuild generated enough increased economic output to pay off about half of 
its direct investment and all of the indirect investment. The investment balance of the program 
after four years, $7,115, is higher than for Youth Corps, which was estimated to have no balance.'® 
This estimate is reasonable, because, on average, Youth Corps spent much less on the education 
and training of disadvantaged young people. The YouthBuild balance is lower than that of the 
other three programs at the end of the follow-up period — $8,934 for Jobstart, $13,194 for Chal- 
leNGe, and more than $20,000 for Job Corps. (See Appendix D for details.) Again, these balances 
are not surprising given that ChalleNGe and Job Corps are residential programs and none of the 
three delivered a substantial short-term community service return. 


At the end of four years, YouthBuild’s prospects for paying off its investment balance 
and providing a positive economic return are good, but far from certain. As shown in Table 3.2, 
there were significantly more young people in the program group, compared with the control 


‘4Fringe benefits were estimated based on the national compensation survey conducted by the U.S. Bureau 
of Labor Statistics (U.S. Department of Labor, Bureau of Labor Statistics, 2016c). Average wages are presented 
in Table 3.2. 


1SJastrzab, Masker, Blomquist, and Orr (1996). 
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group, who were working, and working in jobs paying more than $10 per hour at the time of the 
four-year survey interview. As a result, the self-reported earnings of the program group are sub- 
stantially higher than those of the control group. 


In addition, as discussed in Chapter 3, significantly more young people in the program 
group had high school diplomas or equivalency credentials, vocational certificates, and associates 
degrees. The program group might maintain its educational credential edge over the control group 
going forward, given the continued large effect on high school equivalency credential receipt. 
Although the differences are small and not statistically significant, more young people in the pro- 
gram group had also enrolled in college at the end of the four-year period, and more of them 
reported that they would probably attend college in the future. (See Appendix Table C.3.) In ad- 
dition, about a third of participants in the program group still have unspent education awards to 
cover tuition and fees. '° 


Long-Term Return 


The value of YouthBuild as a public investment — as well as the value of other programs 
serving disadvantaged young people — depends on its long-term impacts. In theory, the addi- 
tional human capital acquired by program group members should produce steadier employment 
and higher wages. In practice, these results may not occur. In the evaluation of Job Corps, despite 
significant increases in GED credential receipt, postsecondary certificates, and initial employ- 
ment and earnings, the impacts on employment and earnings did not persist.” Other programs 
targeted to disadvantaged young people also have produced gains in postsecondary credentials 
and short-term earnings, but failed to produce longer-term improvements. '® One exception is Ca- 
reer Academies, a different (high school-based) model serving enrolled high school students. It 
had no effect on high school completion or postsecondary school enrollment, but had significant 
impacts on short-term employment and earnings. The labor market gains grew in Years 5 through 
8, resulting in an average earnings impact over eight years of about $30,000.'? Researchers hy- 
pothesized that the program’s use of “career awareness and development activities” (tasks em- 
phasized by YouthBuild as well) may have contributed to the earnings gains. 


16 total of 722 participants in the program group received education awards at 26 of the programs affiliated 
with AmeriCorps and included in the benefit-cost analysis. This total represents 32.8 percent of the program 
group members who ever participated in YouthBuild. The average award was $1,871, and only a small fraction 
of the awards’ value had been used at the end of the four-year observational period. Many colleges match the 
awards, doubling their value to students. Data on the awards at programs in the evaluation were supplied by 
YouthBuild USA. 


‘7Schochet, Burghardt, and McConnell (2008). 
18Bloom, Thompson, and Ivry (2010). 
‘9K emple (2008). 
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In the general population, the expectations for increased earnings are provided by exten- 
sive research on the return to obtaining postsecondary education credits and credentials.*° For 
example, a review of recent longitudinal studies of the effects of community college on subse- 
quent earnings found that associates degrees increase average earnings five to nine years after 
college enrollment by 18 percent for men and 26 percent for women. The average return in cer- 
tificate receipt is 7 percent for men and 10 percent for women, and the return for credits (no degree 
or certificate) is smaller and depends heavily on the field in which they are earned.*! Looking 
specifically at the experience of young men, researchers estimate that completing two years of 
community college, relative to having a high school diploma, adds 11 to 12 percent to annual 
earings.” 


In this context, how have evaluations addressed the long-term effect on earnings in as- 
sessing program investments? For Job Corps, the benefit-cost analysis used extrapolation meth- 
ods consistent with earlier analyses and concluded the program was a good investment.” Long- 
term follow-up data, showing a reduction in earnings effects, suggested that this initial assessment 
was overly optimistic.** For ChalleNGe, the benefit-cost analysis assumed that the long-term im- 
pacts on earnings would mirror the return to postsecondary education for young people.* In 
Jobstart and Youth Corps, the evaluations made no effort to predict long-term impacts. 


Benefit-Cost Analysis 


The benefit-cost analysis determines estimates of net benefits and costs over the four-year obser- 
vational period, and an estimated range of future benefits, to address a focal question: What is the 
net value of the YouthBuild investment for participants, taxpayers, and society? Young people 
make an investment by participating in YouthBuild, forgoing labor market income to perform 
community service and advance their education and training. Taxpayers (everyone else in soci- 
ety) make a different investment, footing the bill for the program in exchange for community 
housing in the short term and greater contributions from participants (in terms of increased tax 


Belfield and Bailey (2017); Oreopoulos and Petronijevic (2013). 

*1Belfield and Bailey (2017) averaged estimates made by studies in Kentucky, Michigan, North Carolina, 
California, Ohio, Virginia, Washington, and Arkansas. 

2Marcotte, Bailey, Borkoski, and Kienzl (2005). The authors used the National Education Longitudinal 
Study of 1988 (NELS:88/2000) database to estimate the contribution of various levels of postsecondary educa- 
tion on earnings in 1999 (controlling for work experience, urbanicity, ethnicity, high school dropout or GED 
credential status, and other demographics), eight years after the sample was scheduled to graduate from high 
school. 

?3McComnell and Glazerman (2001). 

*4Schochet, Burghardt and McConnell (2008). 

*5Perez-Arce, Constant, Loughran, and Karoly (2012). 
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payments) and self-sufficiency (in terms of decreased reliance on public assistance and services) 
in the longer term. The view of society as a whole — that is, the overall economy — ignores 
transfers between participants and taxpayers (stipends, education awards, taxes, and so on), which 
means that benefits and costs all constitute net changes in the use of resources and economic 
output. 


The research team made estimates of benefits and costs in the analysis using estimation 
methods largely consistent with the benefit-cost analyses of programs such as Job Corps and 
ChalleNGe.*° (See Appendix D for a detailed discussion of benefit and cost estimation.) Thus, 
the estimates can be used not only to assess the YouthBuild investment, but also to compare 
various aspects of the return on investment in YouthBuild with those of other programs with 
similar objectives. The data from the Youth Corps and Jobstart evaluations are not adequate for 
this comparison. 


The results of these cost and benefit calculations are shown in Table 5.4. The first entry 
in the table is the value of construction work that YouthBuild performed for local communities, 
which was described earlier in the chapter and is discussed in more detail in Appendix D. The 
value of $7,867 is a benefit to communities (taxpayers) and to society. Below this entry are esti- 
mates of YouthBuild’s effects on earnings and fringe benefits. The program group earned $155 
less than the control group over the four-year follow-up period, an estimate obtained by summing 
the impacts on earnings for Years 1 through 4 (using records data). If estimated fringe benefits 
are included, the difference is $194. The table shows this difference as a loss to participants. Tax- 
payers are unaffected by these earnings, so there is no benefit or cost to them, and the participants’ 
loss carries over to the perspective of society as a whole (which sums the benefits and costs to 
participants and taxpayers). 


Because of the significant impacts on secondary and postsecondary credentials, there is 
reason to think the measured program-control difference in earnings in Year 4 of $251 (although 
not statistically significant) might be larger in future years. Research indicates that the full impact 
of credentials (especially lengthier certificates and college degrees) may take 18 or more quarters 
(from the time of community college enrollment) to unfold.”” 


The best evidence of what earnings impacts in Year 5 and beyond will be is provided by 
the 48-month survey, which asked about current earnings at the very end of Year 4. The statisti- 
cally significant difference of $33 per week was measured after the program and control group 


*6For details of these studies, see Schochet, Burghardt, and McConnell (2006) and Perez-Arce, Constant, 
Loughran, and Karoly (2012). 


?7Minaya and Scott-Clayton (2017). 
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Table 5.4 
Benefits and Costs of YouthBuild by Perspective (2017 Dollars) 


Category ($) Participants Taxpayers Society 


Value of community construction 0 7,867 7,867 
Measured earnings and fringe benefits -194 0 -194 
Future earnings and fringe benefits 4,461 to 53,284 0 4,461 to 53,284 
Public assistance and taxes -8,202 to -614 614 to 8,202 0 
Additional education costs 0 -317 -317 
Additional costs of criminal activity ) -108 -108 
Net cost of YouthBuild 2,529 -12,820 -10,291 
Net value (benefits minus costs) 6,182 to 47,417 -4,764 to 2,824 1,418 to 50,241 


SOURCES: MDRC calculations using cost data from 54 YouthBuild sites, the Tax Policy Center, the Institute 
on Taxation and Economic Policy, the Federal Reserve Bank of St. Louis, the National Center for Education 
Statistics, the U.S. Bureau of Labor Statistics, and responses to the 48-month survey. 


NOTE: Cost estimates were adjusted for inflation using the gross national product implicit price deflator. 


postsecondary education and labor market behavior had stabilized. Although study participants 
were still relatively young at the four-year mark, postsecondary school enrollment had reached 
its lowest level since shortly after random assignment for both groups (see Chapter 3, Figure 3.1) 
and the trajectory of increased employment for both groups had somewhat flattened (see Chapter 
3, Figure 3.2). In addition, the survey-reported impacts on postsecondary school enrollment and 
on jobs paying more than $10 per hour at 48 months are about the same, suggesting that the effect 
of program-induced postsecondary educational achievements may be reflected in the earnings 
difference. Thus, the second phase of return on investment appears to conclude around the end of 
the fourth year of follow-up. 


While the research team measured this difference in earnings at a suitable point in time, 
and the difference is consistent with research-based expectations, it is not clear whether the impact 
will be sustained over time. As discussed above, there have been multiple cases in which initia- 
tives achieved differences in credentials and short-term earnings, but not in longer-term earnings. 
Thus, the team made alternative estimates of YouthBuild’s future impacts on earnings and fringe 
benefits for this analysis. All the estimates are calculated as the sum of earnings gains over the 
working lives of participants, discounted to reflect 2017 dollars using a 3 percent discount rate. 
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e Pessimistic Estimate. One estimate, $4,461, was made by assuming that the 
measured program-control difference in weekly earnings at the end of Year 4 
declines following the pattern observed in the extended Job Corps follow-up. 
An estimate of similar magnitude results if it is assumed that the measured 
difference in weekly earnings lasts for one year and then goes to zero. 


e Optimistic Estimate. A second estimate, $53,284, was calculated assuming 
that the measured earnings difference at the four-year mark is maintained over 
the working lives of sample members. An even more optimistic estimate could 
be made by assuming the measured difference grows, as it did in the evaluation 
of Career Academies and in many studies of the return on community college 
degrees, credentials, and credits. The chance of such growth in earnings would 
increase if the YouthBuild program group cashes in more education award 
chits in order to complete additional college degrees. 


The two approaches each yield positive estimates of future earnings and fringe benefits, 
but their different magnitudes produce different overall conclusions. Neither of the estimates is 
clearly superior to the other. 


The other estimates in Table 5.4 correspond to the following benefit and cost elements in 
the analysis: 


e Public assistance and taxes paid. Estimates of public assistance and tax pay- 
ment differences between the program and control groups are imputed using 
government data on Temporary Assistance for Needy Families, Supplemental 
Nutrition Assistance Program, and other cash benefits and estimated effective 
tax rates for the lowest income quintile households.”® 


e Postsecondary education costs. As discussed earlier, the indirect YouthBuild 
investment is $928 per program group member and the net cost is $317 per 
program group member. About a third of the cost of postsecondary education 
is paid by program and control group members in tuition and fees. For program 
group members, part of the cost is offset by the education awards they re- 
ceived. 


e Cost of criminal activity. The evaluation did not find statistically significant 
program effects on criminal activity, but did measure a small positive differ- 
ence in arrests through four years (a difference of 0.7 percentage points that 


8Tax Policy Center (2016); Davis et al. (2013). 
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was not statistically significant). Thus, the analysis attaches a value to this dif- 
ference, but does not value the cost of any long-term increase in crime to tax- 
payers and society. The estimated cost of $108 reflects both criminal justice 
system expenses and victimization costs. 


Appendix D provides further details of the estimates. 


These estimates of the benefits and costs of YouthBuild suggest that, given a plausible 
range of potential long-term impacts on earnings, the net benefit of the program for participants 
ranges from little more than $6,000 to more than $47,000 — or from a modest gain to a lasting 
change in economic well-being. The range in the bottom-line value of YouthBuild for taxpayers 
is much narrower — a net cost of about $5,000 to a net gain of about $2,800. This finding suggests 
both that taxpayers should not count on a high budgetary return, and that the program is not a 
risky investment. Also, to the extent that taxpayers want to help the target population, this roughly 
break-even result for taxpayers indicates that YouthBuild is an efficient vehicle for doing so. 


The value to society as a whole depends, as it does for participants, on the extent to which 
the program group’s greater postsecondary educational achievement produces higher earnings. If 
the Job Corps scenario is repeated, YouthBuild will generate a net social cost. If the measured 
difference in earnings at the end of Year 4 does not decay over time, and the measured difference 
in human capital continues to deliver a “normal” return, the net social benefit may be substantial. 
Indeed, for the broad economy (the social perspective), YouthBuild’s benefit-cost ratio under this 
scenario is about 4:1 (or $50,241:$12,820) and the internal rate of return is 20 percent. 


Comparison of Results to Job Corps and ChalleNGe 


Figure 5.2 presents the results of the analysis, alongside the corresponding estimates for the Job 
Corps and ChalleNGe, with all results expressed in 2017 dollars. The costs and benefits shown 
in the chart are calculated from the perspective of society as a whole. Only estimates based on the 
respective evaluations’ observational periods are provided, because the studies used different ap- 
proaches to estimate benefits and costs beyond the follow-up period. The methods used to pro- 
duce the estimates shown are similar. 


The direct investment in YouthBuild is larger than that in ChalleNGe and smaller than 
that in Job Corps. The latter is true because the estimated cost of education and training services 
accessed by the control group is lower for Job Corps than for YouthBuild. 


For YouthBuild there is a further, indirect investment in postsecondary education, while 
in Job Corps the estimated non-Job Corps spending in postsecondary education on the program 
group is less than for the control group. This is understandable given that Job Corps is geared 
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Figure 5.2 


Comparison of Benefit-Cost Results 
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Job Corps -$13,471 $237 -$1,877 $2,704 $504 -$11,902 
= ChalleNGe -$11,552 -$5,026 -$1,830 $2,537 $741 -$15,131 


SOURCES: MDRC calculations using cost data from 54 YouthBuild sites; the Tax Policy Center; the Institute on Taxation and Economic Policy; the 
Federal Reserve Bank of St. Louis; the National Center for Education Statistics; the U.S. Bureau of Labor Statistics; McConnell and Glazerman (2001); 
Perez-Arce, Constant, Loughran, and Karoly (2012); and responses to the 48-month survey. 


NOTES: Cost estimates were adjusted for inflation using the gross national product implicit price deflator. 
Benefit-cost results are from the perspective of society. 


toward providing participants with vocational certificates rather than preparing them for college 
study. In contrast, ChalleNGe’s indirect investment is greater than YouthBuild’s,”° in part because 
no postsecondary education is included in the ChalleNGe intervention. As discussed earlier in 
this chapter, many local programs in YouthBuild partner with colleges to provide for-credit col- 
lege coursework and CNCS-funded programs provide awards to use toward college tuition and 
fees. The combined direct and indirect investments in YouthBuild are actually lower than both 
Job Corps and ChalleNGe. From this standpoint, the relatively high price tag for YouthBuild 
appears to be misleading. 


The output delivered by YouthBuild in Year 1 offsets about half the cost of the net in- 
vestment in the program. Most of this output results from YouthBuild’s construction of local 
housing and facilities. This contrasts sharply with Job Corps and ChalleNGe, which deliver com- 
munity service on a much smaller scale. This distinction highlights a key comparative difference 
in the YouthBuild investment: Most YouthBuild programs spend a great deal of money and their 
participants devote a large share of their time to building houses. Compared with programs such 
as Job Corps and ChalleNGe, this adds to the cost of the investment and reduces the time partic- 
ipants have for their academic studies. However, it also produces something of tangible value to 
communities as well as a community service experience that, it is hoped, improves young peo- 
ple’s ability to succeed in the future. 


The net output delivered by YouthBuild in Years 2 through 4 is lower than what Job 
Corps and ChalleNGe produced. Compared with Job Corps, this lower output may be partly due 
to more YouthBuild program group members studying rather than working. Having earned many 
more vocational credentials than their counterparts in YouthBuild and ChalleNGe, the Job Corps 
program group members spent less time in postsecondary education study and more time in the 
labor market in Years 2 through 4. However, the ChalleNGe and YouthBuild program groups 
spent similar amounts of time in postsecondary education studies. Differences in economic con- 
ditions facing the groups may also help explain the differences in their employment.*° 


The status of the investment in YouthBuild at the end of four years is better than, but also 
similar to, those in Job Corps and ChalleNGe at the end of their respective follow-up periods. 
None of the three programs achieved social benefits equal to its net cost by the end of four years. 


?°Tn the benefit-cost analysis of ChalleNGe (Perez-Arce et al., 2012), the average cost per full-time student 
at all degree-granting colleges and universities was applied to postsecondary enrollment by the program and 
control groups. This unit cost is more than twice that of two-year institutions, the cost applied to enrollment in 
this analysis. The authors explained that this was done for consistency, because they estimated future earnings 
gains based on average returns on additional enrollment by young people in all (two- and four-year) institutions. 

3°During the time of random assignment in the ChalleNGe evaluation (2005-2007), the unemployment rate 
was 4.4 to 5.4 percent; it was 4.4 to 9.4 percent during the three-year observational period for cohorts in the 
sample. During random assignment in the YouthBuild evaluation (2011-2012), the unemployment rate was 7.9 
to 9.0 percent; it was 4.8 to 9.1 percent during the four-year observational periods for the YouthBuild cohorts. 
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YouthBuild is closest to the break-even point. It will reach this point in about two years if the 
measured program-control difference in earnings is maintained, and it potentially could amass a 
large social return. Job Corps was projected to reach and pass the break-even point, but failed to 
do so. For ChalleNGe to reach the break-even point, the measured difference in weekly earnings 
at the end of follow-up — identical to that of YouthBuild when expressed in 2017 dollars — 
would need to continue for more than six years. Finally, whatever the economic return on the 
investment in YouthBuild turns out to be, it will miss the potential benefits suggested by the 
program’s impacts on civic engagement. 
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Chapter 6 


Conclusion 


The young men and women who signed up for a chance to enroll in YouthBuild reported that 
they did so to get their lives back on track. Most wanted the program to help them get a high 
school equivalency credential, go to college, and find a good job. Although demonstrating moti- 
vation and persistence by signing up, they faced a number of challenges. Nearly all of them did 
not have a high school diploma or equivalency credential, with many having very low reading 
and math skills. Many faced unstable living situations, and a third of them had children. They 
also faced an unforgiving labor market, with high unemployment rates for young people and few 
good jobs for those without a college degree. 


The YouthBuild evaluation was designed to test the effects of the program on the young 
people it served, and the findings after four years show that it led to positive, if modest, effects in 
a number of areas. YouthBuild increased education and training. It led to a sizable increase in 
high school equivalency credential receipt, vocational training enrollment, and college enroll- 
ment. However, the program had only a very small effect on college degree receipt. Most young 
people who enrolled in college because of YouthBuild did so in Years 1 and 2 but did not continue 
to earn degrees. 


YouthBuild also increased employment and earnings, according to one source but not 
another. Young people in the program group were more likely to report having a job at the 48- 
month point and they earned higher wages. Similar increases in work, however, were not con- 
firmed using the unemployment insurance (UJ) records data, although there was a trend over the 
four years toward positive effects on earnings. Survey and records data often show different re- 
sults, particularly for low-income groups who are more likely to have informal jobs, be self-em- 
ployed, or work for employers who may not report their wages. 


The program did not have many effects in areas outside of work and education. It did 
increase civic engagement, mostly in the form of volunteering, although most of this volunteer 
work took place during participation in YouthBuild. It did not lead to changes in other measures 
of youth development, such as self-esteem, future orientation, or feelings of social support, al- 
though these measures are difficult to assess with survey data and difficult to affect over the long 
term. Finally, the program did not affect rates of involvement with the criminal justice system. 


Overall, YouthBuild’s effects were on par or somewhat more positive than effects found 
for other youth programs that have been evaluated with randomized controlled trials. Its effects 
on high school equivalency credential receipt, for example, were a bit smaller than those found 
in Job Corps and ChalleNGe, but its effects on postsecondary enrollment were in between these 
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programs’ effects. Both ChalleNGe and YouthBuild, however, had no or very small effects on 
college degree receipt. 


YouthBuild’s effect on work and earnings were also fairly similar to other programs. Job 
Corps, for example, led to a similar-sized increase in survey-reported employment and a some- 
what smaller increase in weekly earnings, and ChalleNGe led to a slightly larger increase in em- 
ployment and a smaller increase in earnings. YouthBuild had no effects on UI-reported earnings, 
but the study sample may be too small to detect the small positive effect in Year 4. 


A limited benefit-cost assessment shows that YouthBuild is valuable to participants, but 
its net value to taxpayers and society depends on the size of earnings impacts beyond the four 
years covered by the evaluation. Underlying questions are whether the additional postsecondary 
educational enrollment caused by participation in YouthBuild will have the sustained effect on 
earnings it typically has for the general population and whether the positive effects on work and 
earnings observed on the survey at the 48-month signal that the young people were shifting to a 
more positive long-term trajectory. Typical of other youth programs and of other investments, 
such as attending college, YouthBuild made a significant upfront investment in these young peo- 
ple, an investment that may take years to fully pay off. 


Next Steps 


Although it is hard to say whether the effects on work and earnings will grow over time, the 
effects observed so far suggest that the model provides a good starting point, but one that could 
be improved upon. The remainder of this section presents three areas for potential improvement, 
which may help YouthBuild have larger effects on the young people it serves and increase its 
return on investment. 


Screening 


The program’s screening processes are designed to ensure that young people who enter 
the program have a good chance of completing it. As a result, young people who made it through 
the screening process and into the study were likely more motivated and persistent at the time 
they applied than the typical young person who has not completed high school. This level of 
motivation could be seen in the control group’s high rate of participation in services during the 
follow-up period. 


One implication of the screening is that it provides a certain context for interpreting the 
program’s effects. The evaluation is not estimating YouthBuild’s effects relative to a group who 
received no services, but rather estimating effects for program participants compared with other, 
similarly motivated young people who sought out alternative services in their communities. 
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Another implication relevant to program improvement, however, is that programs may 
want to consider conducting less-intensive screening. During site visits by the research team, 
many staff reported the importance of “readiness for change” when considering interested appli- 
cants. While the focus on recruiting “committed” applicants may help improve outcomes, it may 
be muting the net impacts or value added measured in this study. In addition, many YouthBuild 
programs also use basic education test scores to screen out those with very low math and reading 
levels, which probably also screens out young people who dropped out of high school in the early 
grades. Yet, the evaluation found that effects on employment were larger for young people who 
entered the program with lower completed grade levels. In addition, effects were also larger 
among programs that did not screen out applicants based on test scores. 


Serving a broader group of young people, including those with less academic preparation, 
may lead to larger program impacts but would also open up the program to a group with even 
greater needs. Less screening in this area might also make program recruitment easier. When the 
study began, the prevailing assumption was that YouthBuild programs would easily have more 
suitable applicants than they can serve. This assumption led the study team to suggest that for 
every 10 applicants, 6 should be assigned to the program group and 4 to the control group. In 
reality, many programs had difficulty finding enough suitable applicants to meet this ratio, or 
even a more relaxed ratio. A potential limitation of this strategy is that it may undercut the per- 
formance measures to which many programs are held and that undoubtedly influence who is ad- 
mitted. This issue is discussed further below. 


Postsecondary Education 


The YouthBuild model stipulates that programs should promote a “culture” of postsec- 
ondary enrollment, helping students prepare for a range of options, such as certification programs 
and two-year and four-year colleges. The YouthBuild design standards also require programs to 
have postsecondary educational partners. Although most programs provided some type of post- 
secondary educational support for students who were interested, programs’ focus on postsecond- 
ary educational preparation was more varied than for high school diploma or equivalency creden- 
tial preparation. 


Postsecondary educational services ranged from “light touch” to very intensive. For ex- 
ample, almost all programs offered college tours and assistance with filling out college and finan- 
cial aid applications, but the less-intensive programs tended to view these services as something 
young people could engage in as they prepared to leave YouthBuild. The more intensive pro- 
grams provided these and additional services, such as providing direct financial aid, access to 
college classes, dual enrollment, or other opportunities to earn college credit. Programs with 
strong postsecondary educational services also tended to integrate these services into the entire 
program. 
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In one program, for example, participants attended multiple college-readiness work- 
shops, toured multiple colleges, applied for financial aid and scholarships, and applied to the local 
community college. In another program, all participants were enrolled in the local community 
college, since that institution provided all of YouthBuild’s academic services. A part of tuition 
costs were waived while participants were enrolled in the program. 


The evaluation found that the more intensive services, not surprisingly, led to larger ef- 
fects on college enrollment, indicating that all programs could learn what works from the high- 
fidelity postsecondary educational programs. However, while YouthBuild successfully served as 
an access point to postsecondary education, the next challenge is to increase persistence in college 
and degree completion. Even the high-fidelity postsecondary educational programs did not in- 
crease degree completion, although their effects on enrollment lasted longer than effects in the 
other programs. 


The problem of low persistence is not unique to YouthBuild, and community colleges 
around the country struggle with how to increase degree completion among low-income students. 
While it may be beyond the reach of YouthBuild’s services to directly affect college persistence, 
several promising strategies have emerged from recent studies.? There may be lessons from this 
research in terms of the types of institutions with which to partner as the program graduates young 
people, such as those with strong support systems for new students. Perhaps more could also be 
done to secure financial aid, since many low-income students leave school because they lack 
resources. 


Finally, implementing what works requires funding. The programs with high postsec- 
ondary education fidelity were more likely to have funding designated for postsecondary educa- 
tional services. Many of these programs received grants through the Postsecondary Education 
Initiative or the National Schools Initiative, both YouthBuild USA initiatives designed to help 
YouthBuild programs develop stronger connections to postsecondary institutions. 


Employment and Earnings 


YouthBuild led to increases in employment and earnings, in one data source but not an- 
other, although these effects were modest in size. The evaluation found that the strength of career 
development services, as measured by the fidelity score, was not associated with effects on em- 
ployment. But the process study identified career development as one of the most challenging 
components for programs to implement. Most programs did not have staff members dedicated to 
cultivating employers, identifying job openings, and placing people in jobs. Smaller programs 


‘Note that the U.S. Department of Labor (DOL) funding period of performance only counts a program’s 
service to young people for one year after the program ends. 


*See, for example, Scrivener et al. (2015) and Scrivener and Coghlan (2011). 
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especially felt constrained by funding and were unable to devote enough staff time to support 
young people with job searches and job placement. 


The job preparation services in YouthBuild included work-readiness training, intern- 
ships, and job search and job placement assistance. Work-readiness training was designed to pro- 
vide young people with the soft skills needed for work, and included topics such as positive work 
behaviors, résumé development, interview training, and appropriate dress for interviews and jobs. 
Toward the end of program engagement, staff members placed an increased emphasis on job 
search, job development, and job placement. For these services, most programs relied on the 
American Job Center listings, personal connections or word of mouth, and job fairs to help par- 
ticipants find jobs. Most programs that used the American Job Centers did not have close working 
relationships with them, partly because they did not believe that these centers were able to provide 
the kind of intensive, one-on-one support that YouthBuild participants needed. 


Finally, most programs reported placing some participants in construction jobs, and most 
placements were in entry-level jobs with general contractors. Finding construction-related em- 
ployment was challenging during the evaluation period, given that many communities’ housing 
markets were still recovering from the Great Recession’s significant effects on the home con- 
struction sector. For young people not interested in construction-related jobs, the most common 
permanent work opportunities were in retail, grocery stores, food service (primarily large fast 
food chains), maintenance and custodial service, warehouse work, and security. 


Just as YouthBuild programs increased their focus on postsecondary educational services 
in response to the changing labor market, they also made changes to their employment services. 
In recent years, for example, many programs have started offering training in other areas in addi- 
tion to construction, such as in information technology or health care. In 2012, the U.S. Depart- 
ment of Labor (DOL) began giving selected grantees (those who had received prior grants) the 
option of offering a “construction plus” model, in which they provide training for construction as 
well as for other high-demand jobs. 


While training in construction is integral to YouthBuild’s identity and helps it to achieve 
its community service goal by building housing for low-income communities, it is possible that 
exclusively focusing on construction could limit its effects. The downturn in the housing market 
made it very difficult for many programs to provide adequate training opportunities for young 
people and also may have limited programs’ ability to place them in related jobs. It is difficult to 
determine whether the focus on construction was a limitation, since the effects on work and earn- 
ings may have been different in a stronger economy. In fact, the number of construction jobs is 
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projected to grow rapidly over the next several years, suggesting increased opportunity in this 
sector.? 


More recently, with the advent of the Workforce Innovation and Opportunity Act of 2014 
(WIOA), which authorizes funding for YouthBuild, grantees are now required to partner with the 
American Job Centers, to help build relationships with local employers and stay on top of changes 
to the local labor market. In addition, there is increased emphasis on Apprenticeship programs, 
which provide occupational skills training on the job. YouthBuild programs are expected to op- 
erate akin to pre-apprenticeship programs and to develop relationships and pathways for young 
people to enter Apprenticeship programs upon graduation. 


Another path consistent with this focus on apprenticeships, and with an expanded focus 
on postsecondary education, would be for YouthBuild to serve as a bridge to proven occupational 
skills training programs. Examples of sector-based training programs can be found in the Work- 
Advance study.* These programs provided intensive training for jobs in a particular sector — 
such as information technology, health care, or manufacturing; job placement assistance; and help 
with job retention and advancement. The study found that well-implemented programs can lead 
to large increases in work and earnings for the individuals they serve. 


YouthBuild Today 


YouthBuild is a very different program than it was when it began in the 1970s. It has changed 
even in the seven years since the evaluation started and continues to evolve today. Some changes 
have come through the integration of YouthBuild into the broader DOL workforce system, sup- 
ported by the DOL-YouthBuild USA technical assistance partnership. YouthBuild USA as an 
organization has also continued to evolve its focus and mission. 


Within DOL, there is a growing emphasis on partnering with employers and trade unions 
to create apprenticeships for YouthBuild graduates. The Workforce Innovation and Opportunity 
Act (WIOA) guidelines that authorize the DOL YouthBuild program encourage this emphasis, 
and several YouthBuild programs in California have implemented Construction Academies that 
have taken this approach. The initial success of these programs has led to replication in other 
states. YouthBuild USA is also urging programs to forge partnerships with local employers to 
engage them in curriculum design and to facilitate work-based learning opportunities and even- 
tual internships. 


3Henderson (2013). 
4Schaberg (2017). 
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In addition, DOL has expanded the focus of the YouthBuild model to provide occupa- 
tional skills training in a broad array of in-demand industries beyond construction while continu- 
ing to support the development of apprenticeship pathways in these additional industry sectors. 
Under WIOA, YouthBuild programs are also required to be partners of the One-Stop system, 
which creates greater alignment of the services provided to participants and allows YouthBuild 
programs to leverage more resources instead of duplicating efforts. This requirement also helps 
YouthBuild programs to strengthen their employer engagement and outreach to better ensure 
post-exit placements for young people. 


YouthBuild USA also supports the development of additional career tracks through a 
focus on STEM (science, technology, engineering, and mathematics) fields and information tech- 
nology to help its participants compete in today’s economy as well as meet the needs of employ- 
ers. YouthBuild USA is also focusing on training its instructors on learning differences, or what 
strategies work for different types of young people based on their learning styles. Finally, the 
organization is also beginning to strategize about how to better engage graduates, to provide more 
in the way of post-program support. 


As one of the program’s major funders, DOL has made a significant investment in Youth- 
Build. The findings from the evaluation suggest that one area for program improvement may be 
in the area of program funding and assessment. The implementation report documented that fund- 
ing stability was an important issue. Most programs were heavily reliant on the DOL funding 
cycle, in which they competed for and hopefully received a new grant every two years. Not win- 
ning a new grant often led to unstable funding, which affected staff stability and program quality. 
In some cases, it meant shutting down. In fact, 13 of the programs in the study closed their doors 
after serving the cohort of young people enrolled in the study. A longer funding cycle may help 
to improve program quality by increasing program stability and by allowing staff to focus less on 
fundraising and more on program components. 


Program performance is also based on a set of standards, which at the time of this evalu- 
ation were called the common measures, used by most youth programs that received DOL fund- 
ing under WIOA’s precursor, the Workforce Investment Act. Under WIOA, there are six perfor- 
mance indicators used to report program success for youth programs. These measures include, 
for example, the percentage of participants either in education or training or employed at several 
points after program exit, and the percentage of participants who have obtained high school or 
post-secondary educational credentials within one year after program participation (although a 
high school credential is only counted if it is combined with employment). The findings from the 
evaluation suggest that the program might have greater impact if it were to serve less academi- 
cally prepared young people. However, the performance standards may create a disincentive for 
doing so since programs that intentionally serve less-skilled young people would be at a disad- 
vantage relative to other programs when competing for the next DOL grant. 
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Assessing program performance and compliance is important, and the outcomes included 
in the performance indicators do track key program goals. However, earlier research has found 
that program outcomes may not be a good measure of program impacts, even when they are 
“adjusted” for participant characteristics.° For instance, employment rates and college enrollment 
rates may be fairly low for a group of YouthBuild participants who left high school in the ninth 
grade, but the findings presented in this report suggest that these rates would have been even 
lower had the group not participated at all. Ideally, programs would be measured and rewarded 
based on this net effect that they have on the young people they serve. 


°Schochet and Fortson (2012). 
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Appendix Table A.1 


Selected Evaluations of Programs for Young People Without High School Diplomas 


Evaluation (Dates) 


Nonresidential 


American Conser- 
vation and Youth 
Service Corps 
(1993-1996) 


Conservative 
Corps Evaluation 
(2011) 


JOBSTART 
(1985-1993) 


National Job 
Training Partner- 
ship Act (analysis 
of out-of-school 
young people) 
(1987-1994) 


Center for Em- 
ployment Training 
Replication (1995- 
1999) 


Target Group 


Mostly 18- to 25-year- 
old out-of-school 
young people 


Mostly 18- to 25-year- 
old out-of-school 

young people, mostly 
young people of color 


17- to 21-year-old high 
school dropouts with 
low reading levels 


Disadvantaged 16- to 
21-year-old out-of- 
school young people 


Disadvantaged 16- to 
21-year-old out-of- 
school young people 


Program Model 


Paid work experience in communi- 
ty service projects, education and 
training, support services 


Temporary, full-time, subsidized 
work in community service 
projects; basic adult education; 
opportunities to earn college credit; 
case management; job-readiness 
skills 


Education, training, support 
services, job placement assistance 


Education, job-skills training, job 
placement, on-the-job training, and 
support services 


Education and vocational training 


Sample Size and 
Number of Sites 


1,009 young 
people and 4 sites 


2,043 young 
people and 21 
sites 


1,914 young 
people and13 
sites 


5,690 young 
people and 16 
sites 


1,485 young 
people and 12 
sites 


Summary of Results 


Increases in employment and 
decreases in arrests, particularly for 
African-American men 


No significant impacts on employment 
or enrollment in school 


Increases in GED credential receipt; 
few impacts on labor market outcomes 
(except at one site) 


No earnings impacts for women or for 
men who have not been arrested; 
possibly negative impacts for men who 
have been arrested 


Few impacts on employment and 
earnings overall; some impacts for 
younger participants 


c6 


Appendix Table A.1 (continued) 


Evaluation (Dates) 


Target Group 


Program Model 


Sample Size and 
Number of Sites 


Summary of Results 


Residential 


Job Corps 
(1994-2003) 


National Guard 
Youth ChalleNGe 
(2005-2011) 


Disadvantaged 16- to 
24-year-old young 
people 


16- to 18-year-old high 
school dropouts who are 
drug free and not heavily 
involved with the justice 
system 


Employment, education, and training 
in a (mostly) residential setting 


Education, service to community, and 
other components in a quasi-military 
residential setting; 12-month post- 
residential mentoring program 


15,386 young 
people and 110 
sites nationwide 


1,173 young 
people and 10 
sites nationwide 


Earnings and employment impacts in Years 
3 and 4 of the study period; impacts faded 
after Year 4, according to administrative 
data; results appear stronger for older 
participants (those 20 to 24 years old) 


Impacts on GED credential receipt, 
postsecondary enrollment, and employment 
and earnings at the three-year follow-up 
point 


SOURCES: Maynard (1980); Gueron (1984); Jastrzab, Masker, Blomquist, and Orr (1996); Cave, Bos, Doolittle, and Toussaint (1993); Orr et al. (1997); 
Quint, Bos, and Polit (1997); Miller et al. (2005); Schochet, Burghardt, and McConnell (2008); Millenky, Bloom, Muller-Ravett, and Broadus (2011); Price et al. 


(2011). 
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This appendix describes the processes used to select YouthBuild programs to participate in the 
evaluation and describes how random assignment procedures were implemented at each 
program, including a discussion of the effect of random assignment on recruitment, eligibility, 
and enrollment. Next, the appendix discusses the impact analysis model’s specifications, 
including weighting and the handling of missing data, followed by a presentation of selected 
impacts per participant. Finally, the appendix describes analyses of cross-site impact variation 
and survey response bias. 


Site Selection 


Not all programs receiving U.S. Department of Labor (DOL) or Corporation for National and 
Community Service (CNCS) funding at the start of the evaluation could be included in the 
evaluation, either because they were unable to continue providing services during the period in 
which study participants were to be enrolled, or because of other concerns about their suitabil- 
ity. Thus, the first step in the evaluation was to select programs for inclusion. Deciding on the 
total number of programs to include in the impact component of the evaluation required a 
balance of three objectives: (1) maximizing the representativeness of the sample and the 
statistical power of the impact analysis, (2) ensuring high-quality implementation of program 
enrollment and random assignment procedures, and (3) evaluation budget considerations. 
Ultimately, 75 programs were included in the evaluation. Fifty-eight of these were selected 
from the programs awarded grants by DOL in 2011, and 17 were selected from programs that 
did not receive DOL funding in 2011 but did receive funding from CNCS.! The latter programs 
are referred to here as CNCS-funded programs, although they might have received funding 
from other, non-DOL sources.” The programs participating in the study look very similar to all 
programs funded by DOL and CNCS in 2011. 


Selecting DOL-Funded Programs 


DOL awarded grants to 74 YouthBuild programs in May 2011.3 Of these 74 programs, 
3 were deemed to be a poor fit for the evaluation because young people assigned to the control 
group were likely to receive substantially the same services as those in the program group. 
Among these programs were ones that operated in conjunction with the Conservation Corps, 
and ones embedded in charter schools where control group members could remain in the school 


"DOL and CNCS chose to include the CNCS-funded programs in the evaluation in order to examine 
whether DOL-funded programs had different impacts than CNCS-funded programs. 


2A number of these programs subsequently received funding from DOL as part of the 2012 funding cycle. 


3An additional two programs received funding to supplement their March 2011 grants. These two pro- 
grams were not considered part of the May class of grantees. 
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and also receive some type of vocational training. Including these programs in the evaluation 
would not have provided a true test of YouthBuild’s effects, since the program and control 
groups would have received nearly identical services. The final sample frame for selection of 
DOL-funded programs thus included 71 programs. 


Given budget constraints, 60 of these programs were selected to participate in the eval- 
uation using probability-proportional-to-size sampling. Each program had a probability of 
selection that was proportional to its expected enrollment in a given program year. This method 
gave each YouthBuild slot (or young person served) an equal chance of being selected for the 
evaluation, meaning that the resulting sample of young people who enrolled in the study should 
be representative of the young people served by these programs. All of the 60 selected programs 
were required by DOL to participate in the evaluation. Of these, however, the study team 
determined during initial discussions with program staff members that 2 programs would be 
unable to enroll any study group participants during the intake period. The final sample of DOL 
programs was thus 58. 


Selecting CNCS-Funded Programs 


CNCS funds programs through its National Direct grant to YouthBuild USA. Forty 
YouthBuild programs received CNCS grants but not DOL funding in 2011. After reviewing the 
available information and conducting phone calls with each of the 40 programs, the evaluation 
team determined that many of these programs, particularly those receiving small CNCS grants, 
were likely to shut down in 2012 or not enroll young people during the study enrollment period. 
For this reason, the study team, along with DOL’s Employment and Training Administration 
and CNCS staff members, opted to select the 24 programs that received CNCS grants of at least 
$95,000 in 2010. Of these 24 programs, 4 subsequently determined that they would shut down 
or otherwise be unable to enroll new participants during the intake period. An additional 3 
programs were deemed to be unsuitable for the evaluation because they operated in areas where 
control group members would be very likely to receive services similar or identical to those 
received by the program group. The resulting sample of CNCS programs was thus 17. 


“According to their grant proposals, the 3 excluded programs planned to serve a total of 133 young people 
in a given program year. The other 71 programs planned to serve a total of 3,171 in a given program year. 
Since the excluded programs accounted for only 4.1 percent of the expected enrollment among DOL-funded 
programs, the study team’s ability to extrapolate the study findings to all DOL-funded programs was not 
compromised. 
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Developing and Implementing Random Assignment Procedures 


Once YouthBuild programs were selected for participation in the random assignment study, the 
study team visited each of them to meet with its leaders and program staff members to further 
explain the study, answer questions, and begin developing plans for the random assignment of 
young people. 


The study team was flexible about when random assignment was conducted, relative to 
the programs’ recruitment activities. Random assignment could be conducted before, during, or 
after Mental Toughness Orientation (MTO). Decisions about the timing of random assignment 
were made in partnership with the program, with the goal of conducting random assignment 
after the point in the recruitment process when a program experienced the largest drop-off, so as 
to maximize the possibility that young people in the program group would ultimately enroll in 
YouthBuild. It was also important, however, to ensure that random assignment was not placed 
so late in the process that the control group would have experienced a significant portion of the 
program. For example, the team avoided placing random assignment toward the end of a 
lengthy MTO. 


Once the study team and a program developed a random assignment plan together, the 
team customized a research procedures manual for that program’s staff members. This manual 
detailed the research design and the steps required of program staff members at each step, from 
outreach through enrollment. Members of the study team usually conducted another site visit to 
train all staff members in these procedures, including the procedures for entering data into the 
MDRC random assignment system, which included basic identifying information about study 
participants such as their names and Social Security numbers.° 


The study team was in communication with programs regularly to monitor their pro- 
gress toward their outreach and recruitment goals, and to monitor the drop-off from application 
to enrollment. If a program was having challenges with recruitment, the study team worked 
with that program to brainstorm ways to improve its numbers. For example, a program might do 
more outreach, delay the start of certain processes (such as MTO), or engage in multiple rounds 
of recruitment. The study team offered advice and support throughout; YouthBuild USA 
coaches and others were also helpful advisers when programs were experiencing challenges. 


Each local program was given a number of “wild cards” that it could use to allow certain applicants it 
selected to bypass random assignment and be allowed to participate. This option was used, for example, when 
a young person’s situation was particularly compelling or when a family member was already a YouthBuild 
participant. Each program was allowed to use 5 percent of its program slots for wild cards. The minimum each 
program received was one wild card. 
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Seventy-two programs successfully completed random assignment at least once during 
the evaluation’s enrollment period of August 2011 to January 2013. The study team allowed 37 
programs that had difficulty reaching their recruitment targets to enroll young people without 
going through random assignment for at least one enrollment cycle, sometimes several. Pro- 
grams might request to bypass random assignment when not enough applicants were present on 
the day of random assignment, when they felt they needed to focus on meeting their DOL or 
CNCS grant recruitment benchmarks, or when they were experiencing significant delays in 
starting their program cycles because they could not recruit enough young people. Three 
programs were never able to conduct random assignment due to low recruitment numbers. 


Appendix Table B.1 presents the baseline characteristics for the full sample and then 
the program and control groups created through the random assignment process. As would be 
expected with random assignment, there are few differences between the two groups with 
respect to these baseline characteristics. 


The Analysis Model 


The basic estimation strategy is to compare average outcomes for the program and control 
groups. Regression adjustment in a linear regression model increases the power of the statistical 
tests. 


The impact analysis used the following model: 


Yq =a + BPy + OXy + yj + ey, 


Where Yj; is the outcome of interest for sample member i in site j, 
a. is the intercept of the regression, 
P; is adummy for membership in the program or control group, 


6 is the set of regression coefficients for the background characteristics X; (including 
age, gender, applied to YouthBuild at a CNCS site, highest grade completed, race/ethnicity, 
whether or not a young person was a parent, and high school diploma or equivalency credential 
completion) for sample member j in site j, 


y are program fixed effects to account for varying random assignment ratios by site, 
and ej is the random error term for sample member i in site j. 


For the analysis of survey outcomes, weights were added to the model to account for 
varying selection probabilities by cohort and research group. 
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Appendix Table B.1 


Baseline Characteristics by Research Group 


Full YouthBuild Control 
Characteristic (%) Sample Group Group 
Age 
16-18 years old 33.0 34.2 30.3* 
19-21 years old 46.3 45.5 48.0* 
22 years old or older 20.7 20.3 21.6* 
Gender 
Male 64.1 63.9 64.7 
Racelethnicity* 
Hispanic or Latino 14.6 14.6 14.4* 
White, non-Latino 15.3 15.6 14.6* 
Black, non-Latino 62.9 62.9 63.1* 
Other? 6.0 6.0 5.9% 
Not specified 1.1 0.8 1.8% 
Has a child 30.0 29.0 32.1** 
Highest grade completed* 
6th or lower 0.4 0.4 0.4 
7th 1.0 1.0 0.9 
8th 7.5 7.7 6.8 
9th 18.6 18.9 18.0 
10th 26.2 26.4 25.6 
11th 34.9 34.2 36.5 
12th 10.0 9.8 10.4 
Has a high school diploma or equivalency credential 9.2 8.8 10.0 
Has a diagnosed disability (learning or physical 10.6 11.1 9.5 
Housing status 
Lives with family 61.0 61.7 59.5 
Owns/rents apartment, room, or house 15.2 15.2 15.2 
Is staying at someone’s apartment, room, or house 15.7 15.2 16.8 
Is staying with foster guardian/is in foster system 0.6 0.7 0.4 
Lives in a halfway house/transitional house 1.2 1.0 1.5 
Is in residential treatment 0.3 0.3 0.2 
Is homeless 3.0 2.7 3.5 
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(continued) 


Appendix Table B.1 (continued) 


Full YouthBuild Control 
Characteristic (%) Sample Group Group 
Who suggested you apply to YouthBuild? 
Family member or relative 29.8 29.0 31.3 
No one 32.5 32.9 31.6 
School counselor, truant officer, teacher, principal 4.3 4.3 4.1 
Friend 20.7 20.6 20.7 
Other 9.7 9.9 9.2 
Reasons for applying to YouthBuild 
High school equivalency credential 87.7 88.0 87.0 
College 63.1 62.6 64.4 
To get life on track 88.2 88.5 87.6 
Job 84.6 85.1 83.5 
Training 67.2 66.7 68.2 
Friends 7.0 7.4 6.2 
Because of children or the need to support family 1.5 1.5 1.6 
Other 4.4 4.6 4.0 
Sample size 3,929 2,700 1,229 


SOURCE: Calculations based on the YouthBuild baseline data form. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 


a . . 
Categories are mutually exclusive. 


Other includes Hawaiian Native or other Pacific Islander, Asian, American Indian or Alaskan, and 
responses of multiple races/ethnicities. 


“This information is missing for some sample members. 


For missing baseline covariates, the study team imputed the sample mean, and the 
model includes dummy variables indicating imputation. Observations with missing values for 
an outcome variable were dropped from the impact analysis for that outcome. Missing 
values for outcome variables were not imputed. 
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Impacts per Participant 


Appendix Table B.2 presents selected impacts per participant, which are sometimes referred to 
in other research as “treatment-on-the-treated” effects. Impacts per participant represent the 
effects of the program on those young people in the program group who actually participated in 
YouthBuild. The study team estimated impacts per participant by dividing the effects on the full 
program group by the fraction of the program group who participated in YouthBuild, based on 
study participant responses to the 12-month survey. 


Cross-Site Impact Variation Analysis 


As described in Chapter 4, it is possible that some YouthBuild programs performed better than 
others relative to their counterfactual alternatives. Using an approach developed by Bloom, 
Raudenbush, Weiss, and Porter,® the study team estimated adjusted empirical Bayes site-level 
program effects for each location, as well as the cross-site standard deviation of program effects. 
Then, the team computed a Q-statistic, which tests whether the estimated cross-site standard 
deviation of effects is statistically significant. To produce the site-level estimates, the team used 
a two-level hierarchical linear model with fixed site-specific intercepts and random site-specific 
program assignment effects. The model also controls for the individual baseline covariates used 
in the full-sample impact model. As noted in Chapter 4, the analysis found statistically signifi- 
cant variation in program effects across sites for each of the five key outcomes. Variation was 
statistically significant at the 1 percent level for each outcome. 


Response Analysis for the 48-Month Survey 


The YouthBuild 48-month survey provides information about the YouthBuild sample members 
on topics such as participation in training and education, employment and job characteristics, 
youth development, and other outcome measures. Since the survey was administered to a subset 
of the YouthBuild sample, it is necessary to assess the reliability of impact results for the survey 
sample in two ways. First, the results for the survey sample may or may not generalize to (or be 
representative of) the full sample because (1) only a subset of the YouthBuild sample was 
selected to be interviewed and (2) individuals who responded to the surveys may be different 
from those who were selected for the survey but did not respond. Second, the failure of some 
sample members to respond to the surveys may compromise the validity of the impact esti- 
mates, particularly if the program and control groups responded to the survey at different rates. 


5Bloom, Raudenbush, Weiss, and Porter (2017). 
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Appendix Table B.2 
Impacts Per Participant for Selected Outcomes 


YouthBuild Control Difference Impact Per 


Outcome Group Group (Impact) Participant 


Education and training (%) 


Earned high school equivalency credential 34.5 23.5 11.0 14.8*** 
Ever enrolled in vocational school 32.9 21.7 11.2 15.1*** 
Received trade license/training certificate 5.3 3.4 1.9 2.5* 

Ever enrolled in postsecondary courses 26.9 21.7 5.3 ope lias 


Work and earnings 


Currently employed (%) 50.9 46.4 4.5 6.1** 
Average weekly earnings ($) 206.7 174.1 32.6 44.0*** 
Average earnings in Year 4 ($) 6,980 6,729 251 339 


Youth development 
Civic engagement? (%) 94.3 90.6 3.7 5/0*** 


Self-esteem score? 3.3 3.3 0.0 0.0 


Criminal justice involvement (%) 


Arrested since random assignment 32.0 31.3 0.7 1.0 
Convicted since random assignment 19.8 17.4 2.5 3.4 
Sample size (total = 2,721) 1,784 937 


SOURCES: MDRC calculations using data from the National Directory of New Hires and responses to the 
48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Impact per participant refers to the difference in program and control group means divided by the 
participation rate (0.74). 


“Civic engagement" is defined as at least one of the following: volunteering, being registered to vote at 
the time of the survey, having voted, or being involved in politics or local community activities. 

Self-esteem is measured using the 10-item Rosenberg Self-Esteem scale. Response categories range 
from 1 = "strongly disagree" to 4 = "strongly agree," where higher scores indicate higher levels of self- 
esteem. Responses to the 10 items are averaged. 
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This section presents a description of the survey fielding efforts, assesses whether sur- 
vey findings can be generalized to the full research sample, and assesses the survey’s validity 
for estimating program impacts. Overall, the results suggest that the survey samples provide 
valid estimates of the program’s effects that can be generalized to the research sample. 


Sample Selection and Survey Administration 


The research sample includes 3,929 sample members. Due to budget constraints, the 
study team could select only 3,436 of the full sample to be interviewed for the survey (that is, to 
be in the fielded sample), as described in Appendix Box B.1. This fielded sample is used for all 
surveys in this evaluation. 


Appendix Box B.1 


Sample Definitions 


Research Sample: All 3,929 sample members who were randomly assigned during the 
sample intake period, which extended from August 2011 through January 2013. 


Fielded Sample: A total of 3,436 sample members were selected for the surveys. 
Respondent Sample: Fielded sample members who completed a given follow-up survey. 


Nonrespondent sample: Fielded sample members who did not complete a given follow- 
up survey for various reasons. For example, because they could not be not located or 
refused to be interviewed.* 


*The nonrespondent sample at the 48-month follow-up point includes 32 deceased sample members, 
99 incarcerated sample members, and 1 sample member in active military service. 


All research sample members who were randomly assigned between August 2011 and 
February 2012 were included in the fielded sample. It was necessary to include all research 
sample members enrolled during those months in the fielded sample because the 12-month 
survey began before the total research sample size was known. When study enrollment ended in 
February 2013, the total research sample was large enough that it was necessary to select a 
subsample of those randomly assigned from March 2012 through January 2013. Specifically, all 
control group members were included and 76 percent of program group members were random- 
ly selected from each program. This sampling plan ensured that each program was represented 
in the survey analysis and helped achieve a more balanced sample of program and control group 
members. 
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Appendix Table B.3 shows baseline characteristics for the research sample (3,929), the 
fielded sample (3,436), and the nonfielded sample (493). Overall, there are very few statistically 
significant differences between the fielded and nonfielded samples. Fielded sample members 
were more likely than nonfielded sample members to have had children when they enrolled in 
the study, but less likely to have had a diagnosed disability. 


The 48-month survey was fielded (administered to survey recipients) between July 
2015 and June 2017. Sample members were asked to complete the survey online, and those 
who did not do so were then called on the phone and asked to complete the survey that way. If a 
sample member still could not be reached, a field representative of the survey firm followed up 
in person. It took slightly different amounts of time to complete the survey online, by phone, or 
in person, but on average sample members completed the survey in less than 35 minutes. 


Characteristics of Respondents and Nonrespondents in the Fielded 
Sample 


Of the 3,436 young people who were chosen to be surveyed, 2,695 completed the 48- 
month follow-up survey, for a response rate of 78 percent; 2,225 completed the 12-, 30-, and 
48-month surveys.’ The program group had slightly higher response rates than the control group 
(the response rate was 80 percent for the program group and 76 percent for the control group). 
Although statistically significant, the difference in response rates between the two research 
groups was small, at 4 percentage points. 


Appendix Table B.4 presents selected baseline characteristics of survey respondents 
and nonrespondents. Some differences are to be expected, since individuals who respond to 
surveys tend to be different from those who do not. Nonrespondents are often people in harder- 
to-reach groups, such as those with lower incomes and greater mobility. The table illustrates 
these types of differences. For example, the respondent sample had a higher percentage of 
women than the nonrespondent sample. This finding is not surprising, as women often respond 
to surveys at higher rates than men.® The respondent and nonrespondent samples for both 
surveys also had different racial and ethnic makeups and different housing statuses. Respond- 
ents were more likely than nonrespondents to have had a high school diploma when they 
enrolled in the study, and were more likely to have applied because they wanted to get their 
lives back on track. 


’The main text of the report and the impact analysis use a broader measure of survey response than the 
analysis presented in this appendix. (For example, 2,721 sample members answered at least some questions on 
the 48-month follow-up survey.) This appendix and supporting analyses use survey completion to define 
“respondents.” Results using the broader measure are similar to what is shown here. 


8Groves (2006). 
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Appendix Table B.3 


Selected Baseline Characteristics of the Fielded and Nonfielded Samples 


Full Fielded Nonfielded 
Characteristics (%) Sample Sample Sample 
Age 
16-18 years old 33.0 32.9 33.7 
19-21 years old 46.3 46.0 48.1 
22 years old or older 20.7 21.0 18.3 
Gender 
Male 64.3 64.2 64.7 
Racelethnicity* 
Hispanic or Latino 14.6 14.3 16.5 
White, non-Latino 15.3 15.4 14.8 
Black, non-Latino 63.0 63.0 63.0 
Other? 6.0 6.0 5.5 
Not specified 1.1 1.2 0.2 
Has a child 33.7 34.2 29.6* 
Highest grade completed* 
6th or below 0.4 0.4 0.6 
7th 1.0 1.0 1.2 
8th 7.6 7.9 5.6 
9th 18.9 19.2 16.7 
10th 26.6 26.2 29.1 
11th 35.5 35.5 35.1 
12th 10.1 9.9 11.8 
Has a high school diploma or equivalency credential 9.3 9.0 10.9 
Has a diagnosed disability (learning or physical 11.1 10.7 13.8* 
Housing status 
Lives with family 63.0 62.8 64.1 
Owns/rents apartment, room, or house 15.7 15.7 15.6 
Is staying at someone's apartment, room, or house 16.2 16.3 15.4 
Is staying with foster guardian/is in foster system 0.6 0.6 0.6 
Lives in a halfway house/transitional house 1.2 1.2 1.1 
Is in residential treatment 0.3 0.2 0.4 
Is homeless 3.0 3.1 2.7 
Other 0.0 0.0 0.0 
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(continued) 


Appendix Table B.3 (continued) 


Full Fielded Nonfielded 
Characteristics (%) Sample Sample Sample 
Locus of Control Score 3.4 3.4 3.3 
Offender status (%) 0.1 0.1 0.1 
Who suggested applying to Youth Build 
Family member or relative 30.7 30.7 30.7 
No one 33.6 34.1 30.1 
School counselor, truant officer, teacher, principal 4.4 4.4 4.6 
Friend 21.3 21.1 23.0 
Judge or someone from the justice system 3.3 3.3 3.5 
Someone else 4.3 4.3 4.2 
Case manager, counselor, mentor, program staff 2.4 2.2 4.0 
Reasons for applying to YouthBuild 
High school equivalency credential 90.9 91.3 88.8 
College 66.0 66.1 65.0 
To get life on track 92.1 92.2 91.2 
Job 88.7 89.0 86.9 
Training 71.4 71.6 70.4 
Friends 7.7 7.8 7.2 
Sample size 3,929 3,436 493 


SOURCE: Calculations based on the YouthBuild baseline data form. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 


percent. 
a . . 
Categories are mutually exclusive. 


Other includes Hawaiian Native or other Pacific Islander, Asian, American Indian or Alaskan, and 


responses of multiple races/ethnicities. 


“This information is missing for some sample members. 


These differences were also tested in a logistic model, in which the probability of re- 
sponse was regressed on a range of baseline covariates. A test of joint significance is statistical- 
ly significant. The differences between the respondent and nonrespondent samples suggest that 
some caution should be exercised when generalizing the survey findings to the research sample. 
However, because the response rate was fairly high (nonrespondents represent just over 20 


percent of the fielded sample), the respondent sample still looks similar to the fielded sample. 
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Appendix Table B.4 


Selected Baseline Characteristics of 


Respondents and Nonrespondents to the 48-Month Survey 


Characteristic (%) Respondents Nonrespondents Full Sample 
Age 

16-18 years old 32.5 34.4 33.0 
19-21 years old 46.2 45.3 46.3 
22 years old or older 21.3 20.2 20.7 
Gender 

Male 61.3 74,7 64.3*** 
Racelethnicity* aot 
Hispanic or Latino 13.8 16.1 14.6 
White, non-Latino 14.8 17.7 15.3 
Black, non-Latino 64.8 56.6 63.0 
Other® 5.6 7.7 6.0 
Not specified 1.0 1.9 1.1 
Has a child 34.5 33.3 33.7 
Highest grade completed* 

6th or below 0.3 0.5 0.4 
7th 1.0 1.0 1.0 
8th 7.9 7.7 7.6 
9th 19.4 18.3 18.9 
10th 26.0 26.9 26.6 
11th 35.2 36.8 35.5 
12th 10.2 8.8 10.1 
Has a high school diploma or equivalency credential 9.5 7.2 9.3** 
Has a diagnosed disability (learning or physical 10.4 11.8 11.1 
Housing status ae 
Lives with family 63.3 60.9 63.0 
Owns/rents apartment, room, or house 15.9 14.8 15.7 
Is staying at someone's apartment, room, or house 16.0 17.4 16.2 
Is staying with foster guardian/is in foster system 0.6 0.6 0.6 
Lives in a halfway house/transitional house 1.0 2.1 1.2 
Is in residential treatment 0.2 0.6 0.3 
Is homeless 3.0 3.5 3.0 
Other 0.0 0.1 0.0 
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Appendix Table B.4 (continued) 


Characteristic (%) Respondents _Nonrespondents Full Sample 


Who suggested applying to YouthBuild 


Family member or relative 30.9 30.1 30.7 
No one 34.6 31.9 33.6 
School counselor, truant officer, teacher, principal 4.2 5.0 4.4 
Friend 21.2 20.8 21.3 
Judge or someone from the justice system 3.0 4.2 3.3 
Someone else 3.9 5.9 4.3 
Case manager, counselor, mentor, program staff 2.2 2.1 2.4 


Reasons for applying to YouthBuild 


High school equivalency credential 91.5 90.4 90.9 
College 66.6 64.3 66.0 
To get life on track 92.7 90.5 92.1* 
Job 89.0 89.0 88.7 
Training 72.0 70.2 71.4 
Friends 7.9 7.4 7.7 
Sample size 2,695 741 3,929 


SOURCE: Calculations based on the YouthBuild baseline data form. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
“Categories are mutually exclusive. 


Other includes Hawaiian Native or other Pacific Islander, Asian, American Indian or Alaskan, and responses 


of multiple races/ethnicities. 
“This information is missing for some sample members. 


Comparisons Between the Research Groups in the Survey Respondent 
Sample 


Although random assignment research designs minimize potential bias, there is a possi- 
bility that the selective nature of the survey response process could result in differences between 
the characteristics of the program group and the control group. If such differences arise, they 
could make the impact estimates derived from the respondent sample less reliable. 


It does not appear that these differences did arise. Selected baseline characteristics for 
program and control group survey respondents are shown in Appendix Table B.5. Overall, the 
two groups look nearly identical. The only statistically significant differences that emerged were 
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Appendix Table B.5 


Selected Baseline Characteristics of Program and Control Group 
Respondents to the 48-Month Survey 


YouthBuild Control 


Characteristic (% Grou Grou 
Age 

16-18 years old 33.4 30.9 
19-21 years old 45.4 47.8 
22 years old or older 21.2 21.3 
Gender 

Male 60.9 62.1 
Racelethnicity* 

Hispanic or Latino 13.4 14.6 
White, non-Latino 15.2 13.9 
Black, non-Latino 64.9 64.7 
Other? 5.6 5.6 
Not specified 1.0 1.2 
Has a child 34.2 35.1 


Highest grade completed* 


6th or below 0.3 0.3 
7th 1.0 1.0 
8th 8.2 7.3 
9th 19.6 19.1 
10th 26.6 25.1 
11th 34.5 36.4 
12th 9.9 10.9 
Has a high school diploma or equivalency credential 9.0 10.6 
Has a diagnosed disability (learning or physical 10.5 10.4 


Housing status 


Lives with family 63.6 62.7 
Owns/rents apartment, room, or house 16.3 15.4 
Is staying at someone's apartment, room, or house 15.4 17.1 
Is staying with foster guardian/is in foster system 0.8 0.3 
Lives in a halfway house/transitional house 0.9 1.1 
Is in residential treatment 0.2 0.1 
Is homeless 2.8 3.2 
Other 0.0 0.0 


(continued) 
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Appendix Table B.5 (continued) 


YouthBuild Control 
Characteristic (%) Group Group 
Who suggested applying to YouthBuild 
Family member or relative 29.6 33.3 
No one 35.9 32.3 
School counselor, truant officer, teacher, principal 4.1 4.4 
Friend 20.9 21.7 
Judge or someone from the justice system 3.1 2.9 
Someone else 4.0 3.5 
Case manager, counselor, mentor, program staff 2.3 1.9 
Reasons for applying to YouthBuild 
High school equivalency credential 92.4 89.7** 
College 65.9 67.9 
To get life on track 93.2 91.6 
Job 89.8 87.3* 
Training 71.5 72.9 
Friends 8.5 6.6* 
Sample size (total = 2,695) 1,765 930 


SOURCE: Calculations based on the YouthBuild baseline data form. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 


percent. 
a . . 
Categories are mutually exclusive. 


Other includes Hawaiian Native or other Pacific Islander, Asian, American Indian or Alaskan, and 


responses of multiple races/ethnicities. 
“This information is missing for some sample members. 


that program group respondents reported more frequently than the control group that they had 
applied to YouthBuild to earn a high school equivalency credential, to get a job, and to make 


friends. 


Consistency of Impacts 


As discussed above, some caution should be exercised in interpreting the results of the 
surveys. Specifically, there are some differences in baseline characteristics between the sample 
members who responded to the survey and those who did not. This section helps to put the 
survey results in context by comparing the impacts estimated based on administrative records 
for respondents with those for nonrespondents. Comparisons using administrative records 
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provide the best estimate of the program’s effects because they use the full program group and 
control group, not a potentially nonrandom subset of survey respondents. If the respondent and 
nonrespondent survey samples have similar impacts estimated using administrative data, it 
would give more credibility to the survey analysis. 


Appendix Table B.6 presents the results, showing impacts for college outcomes using 
National Student Clearinghouse enrollment records and employment and earnings outcomes 
using National Directory of New Hires records. Overall, the impacts were fairly similar for the 
respondent and nonrespondent samples. There were some differences; specifically, the respond- 
ent sample had more positive program impacts on enrollment in a less-than-two-year institution, 
employment in Year 3, and earnings in Year 4, compared with the nonrespondent sample. 
However, tests of joint statistical significance across (1) all educational outcomes and (2) all 
employment and earnings outcomes suggest that survey respondents and nonrespondents did 
not differ across these outcomes. Further, as shown in the table, impacts for the respondent 
sample were very similar to those for the full sample. The largest differences in impacts 
occurred among the earnings outcomes, which was expected due to high variation in earnings 
across the sample. Still, the respondent sample generally resembles the full sample in both 
magnitude and direction of earnings impacts. 
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Appendix Table B.6 


Impacts on Education and Employment Based on Survey Response 
at 48 Months 


Full Respondent Nonrespondent 
Impact Sample Sample Sample 


Attended college since random assignment (%) 


Enrolled in a 4-year institution 1.0 0.8 1.0 
Enrolled in a 2-year institution 7.8 7.7 8.2 
Enrolled in a less-than-2-year institution 0.2 0.4 -0.2* 
Public 8.3 8.1 9.6 
Private 0.7 1.0 -1.0 
Full time 2.8 2.9 1.8 
Received a degree (%) 0.8 0.8 0.9 
Certificate 0.6 0.5 0.7 
Associate's 0.2 0.3 0.0 
Bachelor's -0.1 -0.1 -0.2 
Master's 0.1 0.1 0.0 


Employment since random assignment (%) 


Employed in Year 1 -2.4 -2.7 -1.9 
Employed in Year 2 3.2 3.6 -1.3 
Employed in Year 3 2.1 2.3 -4.8* 
Employed in Year 4 -1.7 -1.6 -8.0 


Earnings since random assignment ($) 


Earnings in Year 1 -314.3 -194.9 -394.3 
Earnings in Year 2 -201.5 -329.5 176.1 
Earnings in Year 3 110.9 20.4 -209.9 
Earnings in Year 4 251.1 458.2 -1107.7* 
Sample size 3,929 2,695 741 


SOURCES: MDRC calculations using data from the National Directory of New Hires and National Student 
Clearinghouse. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance was tested between the respondent and nonrespondent samples. 
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Appendix C 


Other Impacts from the 48-Month Survey 


THIS PAGE INTENTIONALLY LEFT BLANK 


Appendix Table C.1 
Impacts on Service Receipt at 48 Months 


YouthBuild Control Difference 
Outcome (%) Group Group __ (Impact) P-Value 


Ever participated in YouthBuild 82.1 -- -- = ae 


Education-related services 


Ever participated 84.0 75.4 8.7*** 0.000 
High school equivalency preparation 62.7 55.4 ho" 0.000 
Academic tutoring (not related to high 

school equivalency) 24.2 15.5 8.7*** 0.000 
High school diploma preparation courses 34.2 33.2 1.0 0.585 
Standardized achievement test preparation 30.9 22.3 8.6*** 0.000 
College preparation activities® 40.0 24.7 1530 0.000 
Getting help finding financial aid 41.7 27.0 14.7*** 0.000 
Other® 18.8 17.4 1.4 0.399 


Job- or training-related services 


Ever participated 82.0 64.3 17." 0.000 
Job skills training program 55.4 34.9 20;5*** 0.000 
On-the-job training in construction or 

another field 61.7 30.7 31.0*** 0.000 
Job certification program 41.2 22.5 ABs 0.000 
Job search assistance® 67.6 50.6 17.0*** 0.000 
Help applying to a vocational training program® 44.6 26.3 18.3*** 0.000 


Personal development services 


Ever participated 69.1 51.3 WF fae! 0.000 
Help or advice from a mentor 48.4 31.9 16.6*** 0.000 
Life skills training® 41.4 21.7 19. 0.000 
Communication or public speaking training 36.2 14.4 21.8*** 0.000 
Leadership development training 44.9 18.4 26.5*** 0.000 
Health services 34.7 20.0 14.7*** 0.000 
Mental health services 26.0 15.3 10.7*** 0.000 
Working with a case manager 39.3 23.0 16.4*** 0.000 

Sample size (total = 2,721) 1,784 937 


(continued) 
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Appendix Table C.1 (continued) 


SOURCE: MDRC calculations based on responses to the 48-month follow-up survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Includes college awareness or college guidance activities, college preparation or transition programs, 
and preparation for college entrance exams. 


"Includes attending adult education classes, various certification courses, and college attendance. 
“Includes activities such as help filling out an application, writing a résumé, and going to an interview. 
“Includes help with a program application or interview. 

“Includes activities such as parenting skills classes and learning how to balance a checkbook. 
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Appendix Table C.2 
Additional Impacts on Employment and Earnings at 48 Months 


YouthBuild Control Difference 
Outcome (%) Group Group (Impact) P-Value 


Outcomes based on survey responses 


Benefits 
Health insurance 18.9 15.6 3.3** 0.044 
Paid sick and/or vacation leave 18.8 16.6 2.1 0.201 
Retirement or pension benefits 13.6 12.6 1.0 0.500 


Current job industry 


Construction 3.6 3.2 0.4 0.595 
Retail trade 8.2 9.7 -1.5 0.211 
Administ 

Administrative/support/waste management/remediation 5.5 5.4 0.1 0.892 
Health care and social assistance 6.1 4.3 1.7* 0.070 
Accommodation and food service 10.6 9.5 1.1 0.388 
Other 16.2 13.8 2.4 0.118 

Sample size (total = 2,721 1,784 937 


SOURCE: MDRC calculations from responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 
Statistical signficance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Social Security numbers are unavailable for some sample members, who therefore could not be 
matched to the National Directory of New Hires database. 
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Appendix Table C.3 
Additional Impacts on Youth Development Outcomes 


YouthBuild Control Difference 


Outcome Group Group __ (Impact) P-Value 


Leadership role at work, religious group, 


community, school in last 30 days (%) 15.6 16.8 -1.3 0.476 
Involved in the community in the last 30 days? (%) 67.7 68.8 -1.1 0.631 
Currently happy (%) 79.9 80.6 -0.7 0.660 


What the future holds (%) 


Willing to wait for bigger financial rewards 36.1 35.1 1.1 0.605 

Will probably attend college‘ 68.2 65.6 2.6 -- 

Expects to live at least 70 years 81.5 82.3 -0.8 0.632 
Exhibits signs of major depression* (%) 16.9 18.1 -1.1 0.488 
Social support score® 3.0 3.0 0.0 0.679 
Self-efficacy score! 3.0 3.0 0.0 0.876 
Sample size (total = 2,721) 1,784 937 


SOURCE: MDRC calculations based on responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


“Did any of the following activities: helped a community member, attended a community meeting to 
improve community conditions, or served as a positive role model for a kid in the community. 


Based on response to the question, "Would you rather get $80 tomorrow or get $100 three months 
from now?" 


“Among those who have not attended and are not currently attending college. This measure is nonex- 
perimental, so significance level and p-value are not included. 


“Depression is measured using the Patient Health Questionnaire (PHQ-9), a nine-item scale used to 
diagnose depression in clinical settings. Response categories range from 0 = "not at all" to 3 = "nearly 
every day," where higher scores indicate more frequent occurrence of depression symptoms. If the item 
score sum is greater than or equal to 10, the respondent is considered to exhibit signs of major depres- 
sion. 

“Social support is measured using a six-item scale. Response categories range from 1 = "strongly 
disagree" to 4 = "strongly agree," where higher scores indicate stronger social support. The six items are 
averaged. 

‘Self-efficacy is measured using a six-item scale. Response categories range from 1 = "strongly 
disagree" to 4 = "strongly agree," where higher scores indicate higher levels of self-efficacy. The six items 
are averaged. 
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Appendix Table C.4 


Impacts on Other Outcomes at 48 Months 


YouthBuild Control Difference 
Outcome (%) Group Group (Impact) P-Value 
Living arrangement 
Parent's home 38.8 41.3 -2.6 0.204 
Another person's home? 18.2 19.0 -0.8 0.642 
One's own place 35.3 33.2 2.1 0.285 
Incarceration facility 3.7 2.7 1.0 0.199 
Other arrangement” 4.0 3.8 0.3 0.753 
Ever homeless since random assignment 29.7 32.3 -2.6 0.174 
Married or living with spouse/partner 30.3 27.9 2.4 0.205 
Receives government benefits® 56.8 56.2 0.6 0.788 
Has children? 57.2 55.3 1.9 0.293 
Lives with all or some of their children 41.6 39.3 2.3 0.210 
Has noncustodial children 21.0 20.0 1.0 0.521 
Paid child support in last 30 days 4.6 4.2 0.4 0.649 
Charged with a crime since random assignment 28.5 27.8 0.6 0.732 
Involved in a gang fight in the past 12 months 7.1 6.9 0.2 0.863 
Sample size (total = 2,721) 1,784 937 


SOURCE: MDRC calculations based on responses to the 48-month survey. 


NOTES: Results in this table are regression-adjusted, controlling for individual baseline characteristics and 
site fixed effects. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

All outcomes reflect current status unless otherwise noted. 

“Includes living with family other than parents. 

b aoe : 

Includes living in a group home or halfway house, a long-term homeless shelter, or an emergency 
housing shelter (including for domestic violence); living on the street; situations such as college or a 
residential training program; and other situations. 

“Government benefits include Temporary Assistance for Needy Families; the Supplemental Nutrition 
Assistance Program (food stamps); unemployment insurance; the Special Supplemental Nutrition Program 
for Women, Infants, and Children; Supplemental Security Income; foster care payments; and utility 
payment assistance. 

“Includes a person's biological, adopted, foster, and stepchildren, plus any other children for whom the 
person is responsible. 
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Appendix D 


Additional Information on the Benefit-Cost Analysis 


THIS PAGE INTENTIONALLY LEFT BLANK 


Chapter 5 presented the results of the benefit-cost analysis, briefly explaining how the various 
estimates in the analysis were made. This appendix provides additional details on those estimates. 


Program Group Costs per Sample Member 


The first group of estimates in Chapter 5 was of the costs of YouthBuild and other pertinent edu- 
cational, training, and supportive services received by the program group. The estimates of 
YouthBuild program costs were based on the budgetary, expenditure, resource use, and program 
participation data for 54 local programs, a subset of the programs included in the overall evalua- 
tion. The YouthBuild budgetary and expenditure data from the 54 sites cover program years 
2010 and 2011, when most sample members were active program participants. These on-budget 
data cover expenditures charged to the U.S. Department of Labor (DOL) and Corporation for 
National and Community Service (CNCS), as well as to other federal, state and local agencies, 
and to a variety of private funding sources. 


The main data sources are interviews with staff from the 54 programs collected by mem- 
bers of the YouthBuild evaluation team. Management information system (MIS) data for the pe- 
riod provided a check on participation information obtained from the DOL-funded programs. 
Information was collected from some additional programs in the study, but the team ultimately 
excluded these programs from the calculations of costs per program group member because it 
judged the participation estimates it obtained to be unreliable. 


The data on off-budget costs incurred at the local level were obtained during the same 
program interviews. Some off-budget expenses incurred by local programs were associated with 
relatively modest contributions of goods and services by organizations and individuals in the 
community. However, some other off-budget expenses corresponded to important aspects of the 
services received by the program group. For example, much of YouthBuild’s educational com- 
ponent was off-budget for certain local programs. (That is, education resources were contributed 
by a school district or education agency.) 


Two off-budget costs were incurred at the national level. One of these is the cost of the 
education awards earned by participants at YouthBuild programs affiliated with AmeriCorps. 
Each award is based on the number of community service hours a participant records. Participants 
who recorded 1,700 service hours eared a full award: $5,550 (in 2011) that could be spent on 


‘As noted in Chapter 5, the cost analysis was conducted for 54 of the 75 programs in the study for several 
reasons. In most cases, programs were missing cost data because these data were not provided to the site visitor 
at all or in a manner that was usable. In a few cases, the cost data were collected, but data on the number of 
participants were deemed unreliable, meaning that an estimate of cost per participant could not be calculated. A 
separate analysis showed that the 54 programs used for the cost analysis were roughly similar to the remaining 
programs, with the exception that these sites were somewhat less likely to be Affiliates of YouthBuild USA. 
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tuition, fees, and other college expenses. Participants earned a half award of $2,775 for 900 ser- 
vice hours and received smaller amounts for earning a reduced half-time award (675 service 
hours), a quarter-time award (450 service hours), and a minimum-time award (300 service hours). 
YouthBuild USA provided data on these awards for AmeriCorps-affiliated programs. Nobody in 
the program group earned full awards, but about a third earned the other awards — most often 
the reduced half-time and quarter-time awards. Data on use of the awards were not available at 
the time the analysis was conducted. Based on communication with YouthBuild staff members, 
the research team assumed that 10 percent of the awards were used at postsecondary institutions 
by the end of follow-up; only this accrued cost was included in the cost estimates provided in 
Chapter 5. As noted in Chapter 5, recipients could cash the unused awards in years after the fol- 
low-up period, potentially boosting the program-control differences in postsecondary enrollment 
and degrees beyond those observed during the follow-up period. Many colleges match the 
awards, which doubles their value to students, making future enrollment more affordable.* At the 
same time, future use of the awards would add to the estimated off-budget cost used in the benefit- 
cost analysis. 


The estimates shown in Tables 5.1 and 5.3 include both on-budget expenditures and es- 
timated off-budget costs. The research team calculated the costs reported in Table 5.3 by multi- 
plying the unit costs of the service or stipend category in the first four rows of the table by the 
participation rates in the next four rows. For example, the unit cost of education-related services 
reported in the first row, $5,517, is the total one-year cost of YouthBuild education-related ser- 
vices at the 54 programs divided by the number of participants in that year. All on- and off-budget 
costs allocated to the education service category (based on the local program interviews) went 
into the numerator for this unit cost calculation. This unit cost, in turn, was multiplied by the 
fraction of program group participants (across all programs in the evaluation, not just the 54 for 
which unit costs were calculated) receiving education-related services during any follow-up year. 
It is noteworthy that a program group member who did not receive education from YouthBuild, 
but did receive pertinent educational services (for example, high school or GED-related services) 
from another provider, was counted when the participation rate was calculated. Thus, the cost 
attached to such services was the same as if the program group member received the services 
from YouthBuild. 


In Table 5.3, the costs per participant were converted to costs per program group member 
by multiplying per-participant costs by the average participation rate of the program group. These 
are gross costs per program group member. A second calculation presented in the table subtracted 
the estimated costs for control group members, which resulted in the estimated net costs per sam- 
ple member. The estimates of costs per control group member are described below. 


Corporation for National and Community Service (2018). 
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Control Group Costs per Sample Member 


The research team estimated the costs of educational, training, and supportive services per control 
group member in the same way as it did for the program group, except that the unit cost estimates 
were different. The unit cost of education-related services was the simple average of the per-pupil 
expenditures for GED preparation in 2011in California and Massachusetts, two of the three states 
with the most YouthBuild programs in the evaluation. The cost in California was based on school 
districts and community colleges, which together accounted for 97 percent of GED preparation.* 
In Massachusetts, the cost was for the state’s Adult Learner Centers, the primary source of GED 
preparation. 


The unit cost of job- and training-related services was the simple average of the per-pupil 
expenditures on for-credit education courses in community colleges in 2011 in California and 
Massachusetts.° The research team used the cost of these courses because most, but not all, voca- 
tional education courses at community colleges provide academic credit. 


The research team based the unit costs for other services, such as counseling and case 
management, on those of the selected YouthBuild programs. The 18 YouthBuild programs with 
the lowest expenditures on youth development services per participant — that is, the lowest one- 
third of the 54 programs in the evaluation — were used for this purpose. There are two reasons 
for using this cost estimation strategy. First, cost estimates for this type of service were not readily 
available, but were isolated in the data collection for this analysis. Second, using the low-cost 
programs was consistent with the findings of this evaluation’s process analysis, which suggested 
that services provided to control group members were less intensive than in the typical Youth- 
Build program. 


Indirect Costs per Sample Member 


Compared with the control group, a higher percentage of the program group enrolled in college 
and postsecondary technical schools. To estimate the cost of this program effect, the research 
team multiplied the measured differences in enrollment by the unit cost estimates. The difference 
in full-time enrollment over the four-year follow-up period, 2.8 percentage points, was multiplied 
by the estimated cost of full-time study in 2011, which was $5,689 in 2017 dollars.® The differ- 
ence in part-time enrollment over the follow-up period, 8.4 percentage points, was multiplied by 
$1,877, the cost of part-time study in 2017 dollars.” Both unit cost estimates were for study in 


3Legislative Analyst’s Office (2012). 

‘Jones (2017). 

>Legislative Analyst’s Office (2012); Alssid, Goldberg, and Schneider (2011). 
5U.S. Department of Education, National Center for Education Statistics (2016a). 
7 U.S. Department of Education, National Center for Education Statistics (2016b). 
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public two-year colleges. Most enrollment was in two-year colleges, but some was in four-year 
institutions and technical or trade schools that primarily grant certificates. As noted in Chapter 5, 
these unit costs were close to estimates made based on other sources, such as the tuition and fees 
charged by California community colleges. 


Value of Community Construction 


As indicated in Chapter 5, YouthBuild participants constructed affordable housing and other com- 
munity facilities during their enrollment in local programs. The study team made an estimate of 
the value of this housing much the same way as it was done in the benefit-cost analyses of Job 
Corps, Youth Corps, and ChalleNGe. The first step was assigning an appropriate wage and fringe 
benefits rate to the community service hours recorded at evaluation programs, and the second step 
was adding the market value of construction materials used by the programs. 


The research team estimated the number of community service hours per program group 
member as the weighted average service hours per participant in pertinent programs times the 
participation rate for the program group.® Pertinent programs were those included in the benefit- 
cost analysis and the weights were the percentages of the total sample contributed by each pro- 
gram. For 38 of the programs, all of which were affiliated with AmeriCorps, the research team 
obtained the recorded hours during the 2010-2011 and 2011-2012 program years from Youth- 
Build USA. For two unaffiliated programs, the team obtained site-specific estimates. For CNCS- 
affiliated YouthBuild programs with missing hours data, the average hours at programs with 
hours data was used. For other unaffiliated programs, it was assumed there were zero service 
hours. 


The research team multiplied these service hours by an estimate of the wages and fringe 
benefits that would have been paid for low-skilled labor in residential construction. The team used 
the average wage rate for residential carpenter’s helper, $14.52 per hour,? for this purpose. The 
team marked up this wage rate with the fringe benefit rate paid in the construction industry.'° The 
team added the value of construction materials used by the local programs to the estimated value 
of the construction labor. 


The resulting estimate of the value of the residential construction was a conservative 
“supply-side” estimate — that is, it was an estimate of what it would have cost a private contractor 


®The average service hours per participant in each program was for all participants during the two years, 
including some who were not part of the evaluation sample. The participation rate was for the program group. 


°U.S. Department of Labor, Bureau of Labor Statistics (2016d). 
10U.S. Department of Labor, Bureau of Labor Statistics (2016c). 
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to build comparable housing.!! The cost of tools, equipment, and supervision used in the con- 
struction was not included in the estimate, making it a conservative supply-side estimate of the 
value of the construction. 


Estimates of YouthBuild’s Future Impacts on Earnings and 
Fringe Benefits 


As indicated in Chapter 5, the research team made the pessimistic estimate of future earnings by 
assuming that the measured program-control difference in weekly earnings at the end of Year 4 
declined following the pattern observed in the extended follow-up for the Job Corps evaluation. 
Similar to the YouthBuild evaluation, the Job Corps study involved four years of follow-up using 
survey and records data. The extended follow-up period added four to five years of records data 
(depending on the time a sample member was randomly assigned).'* For the pessimistic estimate 
of future earnings, the research team assumed that the survey-measured impact in YouthBuild 
would change (in percentage terms) just as observed in this extended follow-up for Job Corps, 
and subsequently would decline at a rate of 20 percent per year. This scenario of fading earnings 
impacts was plausible, even though YouthBuild’s impacts on high school or equivalency creden- 
tial completion and postsecondary education enrollment were larger than Job Corps’ and there 
was an increase in associate’s degree attainment. 


The research team calculated the optimistic estimate of future earnings assuming that this 
measured earnings difference was maintained over the working lives of sample members. The 
measured difference slightly exceeded the expectations discussed earlier and, assuming the dif- 
ference held up over time, was consistent with expectations for individuals with associates de- 
grees and certificates, but was optimistic for young people enrolled in postsecondary education, 
but without a credential.'? While the measured difference does not decline in future years, its 
dollar value declines annually due to the application of a 3 percent real discount rate. 


Estimates of YouthBuild’s Impacts on Public Assistance 
and Tax Payments 


The research team imputed estimates of public assistance and tax payment differences between 
the program and control groups. For public assistance, the team multiplied the survey-measured 
difference in receipt of government benefits, reported in Appendix Table C.4, by the average total 


‘Kemper and Long (1981) found that such supply-side cost estimates were slightly below demand-based 
estimates of value in similar projects conducted in the Supported Work demonstration. 


'Schochet, Burghardt, and McConnell (2006). 
13Bailey and Belfield (2015). 
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Temporary Assistance for Needy Families (TANF) and Supplemental Nutrition Assistance Pro- 
gram (SNAP) benefits received by TANF recipients in 2011, which was $10,112.'* The measured 
difference in government benefits was not statistically significant, so the team made no estimate 
of effects beyond the observation period. 


For federal tax payments, the research team multiplied program-control differences in 
measured and future (upper- and lower-bound) earnings, and in other pertinent income, by an 
estimate of the effective overall federal tax rate. The Tax Policy Center of the Brookings Institu- 
tion and Urban Institute has computed the average effective rate to be 3.5 percent for all federal 
taxes (including income tax credits) for households in the lowest income quintile.'° The rate ap- 
plied to “expanded cash income,” which was estimated for sample members as earnings, cash 
transfers (TANF and SNAP), and pertinent components of fringe benefits (retirement and health 
insurance). 


For state and local taxes, the average effective tax rate for households in the lowest in- 
come quintile is 11.1 percent, as calculated by the Institute on Taxation and Economic Policy. *® 
However, this rate applied only to estimated program-control differences in earnings and TANF, 
not SNAP or any components of fringe benefits. 


Estimates of YouthBuild’s Impacts on the Costs 
of Criminal Activity 


The evaluation did not find statistically significant program effects on criminal activity, but did 
measure a small positive difference in arrests through four years. Thus, the analysis attached a 
dollar value to this difference — which represented a cost to taxpayers and society — but did not 
value any long-term program effect on crime. As indicated in Table 3.4, the probability of arrest 
was 0.7 percentage points higher for the program group. Consistent with findings from a 2012 
study,” the research team assumed that each individual arrested at least once was arrested a total 
of 1.37 times, and that the estimated criminal justice system and victimization cost per arrest, in 
2017 dollars, was $11,279. The unit cost estimate reflected both criminal justice system expenses 
and victimization costs associated with each arrest. 


14U_S. Department of Health and Human Services, Administration for Children and Families (2013). 
STax Policy Center (2016). 

'6Davis et al. (2014). 

‘”Perez-Arce, Constant, Loughran and Karoly (2012). 
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